[
https://issues.apache.org/jira/browse/PHOENIX-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15553918#comment-15553918
]
Enis Soztutar commented on PHOENIX-3360:
----------------------------------------
Thanks James for taking a look.
bq. In this case, we know that any RPC is due to an index needing to be
updated (hence there's not check)
Not necessarily. RS -> RS communications can happen for other reasons. Meta
updates, stats updates, ACL updates etc. We definitely do not want to use an
index priority unintentionally for stats update or ACL update, etc. This has
been such a fragile area that getting this wrong results in horrible deadlocks
(see HBASE-16773 as a recent example).
bq. The intention is that ServerRpcControllerFactory is only used on the server
The clients running on the same nodes as the servers usually share the same
configuration. Not all deployments are like that, but I can say this is
probably more common that the HBASE_CONF_DIR is shared. This means client
requests that originate from the same nodes that run a server will go to the
wrong queues.
bq. I would love to use HBASE-15816 as we could get rid of all this
RpcControllerFactory nonsense (which we only used because we have no HBase API
to set the priority). This would make it much more clear.
I think we should get that in HBase in any case. The problem with relying on
HBASE-15816 would be that ALL requests should be explicitly tagged with the
correct priority. If we accidentally miss a couple, then we can still deadlock.
Code changes and we add new stuff, new RPCs, etc all the time. I would rather
try to find a future-proof solution rather than relying on code reviews for
making sure that all RPCs are marked.
In HBase, we used to have more server-side computation of the RPC priority
(AnnotationReadingPriorityFunction), which we realized quite costly because the
RPC reader threads have to do the work to find the correct queue for the
scheduler. We are pushing more and more to set the priority at the client side
via RpcControllers. The problem is that the RPCController does not have any
other information about the request other than the table name. In case of
requests to hbase:meta table, or SYSTEM.CATALOG, etc it works perfectly. In
case of Index tables, it fails because we cannot tell whether the table is an
index table by just looking at the name of the table. I was thinking of adding
a HTD cache or something in the RPCController so that the priority can come
from the table descriptor. However, this means that we have to do a master
request to fetch the HTD first. I think that won't work.
> Secondary index configuration is wrong
> --------------------------------------
>
> Key: PHOENIX-3360
> URL: https://issues.apache.org/jira/browse/PHOENIX-3360
> Project: Phoenix
> Issue Type: Bug
> Reporter: Enis Soztutar
> Priority: Critical
>
> IndexRpcScheduler allocates some handler threads and uses a higher priority
> for RPCs. The corresponding IndexRpcController is not used by default as it
> is, but used through ServerRpcControllerFactory that we configure from Ambari
> by default which sets the priority of the outgoing RPCs to either metadata
> priority, or the index priority.
> However, after reading code of IndexRpcController / ServerRpcController it
> seems that the IndexRPCController DOES NOT look at whether the outgoing RPC
> is for an Index table or not. It just sets ALL rpc priorities to be the index
> priority. The intention seems to be the case that ONLY on servers, we
> configure ServerRpcControllerFactory, and with clients we NEVER configure
> ServerRpcControllerFactory, but instead use ClientRpcControllerFactory. We
> configure ServerRpcControllerFactory from Ambari, which in affect makes it so
> that ALL rpcs from Phoenix are only handled by the index handlers by default.
> It means all deadlock cases are still there.
> The documentation in https://phoenix.apache.org/secondary_indexing.html is
> also wrong in this sense. It does not talk about server side / client side.
> Plus this way of configuring different values is not how HBase configuration
> is deployed. We cannot have the configuration show the
> ServerRpcControllerFactory even only for server nodes, because the clients
> running on those nodes will also see the wrong values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
