[
https://issues.apache.org/jira/browse/PHOENIX-7776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18062721#comment-18062721
]
Richárd Antal commented on PHOENIX-7776:
----------------------------------------
The main difference is I think that during the initialization of hbase:meta
HTable we set the rpcControllerFactory to ServerRpcController and during the
HTable.get we use the previusly set one, while in Hbase 3 we create a new
Controller.
I think it happens on the HBase side so the check added in PHOENIX-7370 to
decide if the connection is server-side doesn't get called.
> Fix hanging tests because of Metadata handlerss
> -----------------------------------------------
>
> Key: PHOENIX-7776
> URL: https://issues.apache.org/jira/browse/PHOENIX-7776
> Project: Phoenix
> Issue Type: Task
> Reporter: Richárd Antal
> Priority: Major
>
> With HBase 3 there are a lot of tests not finishing because we ran out of
> metadata handlers configure by METADATA_HANDLER_COUNT_ATTRIB.
> Few tests where it is easily visible are:
> ConcurrentGetTablesIT
> instead of 2 , usually passes with 20 failes with 18
> [https://github.com/richardantal/phoenix-1/blob/dff56a6946585944d1e34dd28de1b135d60ff425/phoenix-core/src/it/java/org/apache/phoenix/end2end/ConcurrentGetTablesIT.java#L74]
> SingleMetadataHandlerIT
> With 1 metadata handler and 1 server side handler it hangs after
> {code:java}
> [RpcServer.Metadata.Fifo.handler=0,queue=0,port=58485]
> client.ConnectionUtils(526): Start fetching meta region location from
> registry{code}
> and there is no log line similar to
> {code:java}
> 2026-02-24T12:13:20,440 DEBUG
> [RpcServer.Metadata.Fifo.handler=0,queue=0,port=58706]
> util.ReflectedFunctionCache(97): Populated cache for
> org.apache.hadoop.hbase.filter.SingleColumnValueFilter in 0ms
> 2026-02-24T12:13:20,440 DEBUG
> [RpcServer.Metadata.Fifo.handler=0,queue=0,port=58706]
> util.ReflectedFunctionCache(97): Populated cache for
> org.apache.hadoop.hbase.filter.BinaryComparator in 0ms {code}
> With 2 metadata handlers and 0 server side handler it passes
> With this change it also passes:
> [https://github.com/richardantal/phoenix-1/blob/dff56a6946585944d1e34dd28de1b135d60ff425/phoenix-core/src/it/java/org/apache/phoenix/end2end/SingleMetadataHandlerIT.java#L57-L61]
> I've looked into it and there is a difference how we check the hbase:meta
> table
> For HBase 2.6
>
> {code:java}
> org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory.getController(ServerRpcControllerFactory.java:58)
> org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory.newController(ServerRpcControllerFactory.java:41)
> org.apache.hadoop.hbase.client.HTable.get(HTable.java:388)
> org.apache.hadoop.hbase.client.HTable.lambda$get$0(HTable.java:371)
> org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
> org.apache.hadoop.hbase.client.HTable.get(HTable.java:371)
> org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1141)
> org.apache.hadoop.hbase.client.HBaseAdmin$8.rpcCall(HBaseAdmin.java:523)
> org.apache.hadoop.hbase.client.HBaseAdmin$8.rpcCall(HBaseAdmin.java:520)
> org.apache.hadoop.hbase.client.RpcRetryingCallable.call(RpcRetryingCallable.java:57)
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104)
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3179)
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3171)
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:520)
> org.apache.phoenix.query.AdminUtilWithFallback.tableExists(AdminUtilWithFallback.java:50)
> org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:1871)
> org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:2556)
> org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:3752)
> org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:1064)
> {code}
> {code:java}
> this.rpcControllerFactory.newController(){code}
> {code:java}
> if (rpcControllerFactory == null) {
> this.rpcControllerFactory =
> RpcControllerFactory.instantiate(configuration);
> } else {
> this.rpcControllerFactory = rpcControllerFactory;
> }
> {code}
>
> For HBase 3
> {code:java}
> org.apache.hadoop.hbase.ipc.controller.ClientRpcControllerFactory.getController(ClientRpcControllerFactory.java:40)
> org.apache.phoenix.compat.hbase.CompatRPCControllerFactory.newController(CompatRPCControllerFactory.java:36)
> org.apache.hadoop.hbase.client.AsyncRpcRetryingCaller.<init>(AsyncRpcRetryingCaller.java:91)org.apache.hadoop.hbase.client.AsyncSingleRequestRpcRetryingCaller.<init>(AsyncSingleRequestRpcRetryingCaller.java:63)
> org.apache.hadoop.hbase.client.AsyncRpcRetryingCallerFactory$SingleRequestCallerBuilder.build(AsyncRpcRetryingCallerFactory.java:166)
> org.apache.hadoop.hbase.client.AsyncRpcRetryingCallerFactory$SingleRequestCallerBuilder.call(AsyncRpcRetryingCallerFactory.java:175)
> org.apache.hadoop.hbase.client.RawAsyncTableImpl.get(RawAsyncTableImpl.java:233)
> org.apache.hadoop.hbase.client.RawAsyncTableImpl.lambda$get$6(RawAsyncTableImpl.java:245)
> org.apache.hadoop.hbase.client.ConnectionUtils.timelineConsistentRead(ConnectionUtils.java:449)
> org.apache.hadoop.hbase.client.RawAsyncTableImpl.lambda$get$7(RawAsyncTableImpl.java:244)
> org.apache.hadoop.hbase.trace.TraceUtil.tracedFuture(TraceUtil.java:86)
> org.apache.hadoop.hbase.client.RawAsyncTableImpl.get(RawAsyncTableImpl.java:243)
> org.apache.hadoop.hbase.client.TableOverAsyncTable.get(TableOverAsyncTable.java:188)
> org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:601)
> org.apache.hadoop.hbase.master.TableStateManager.readMetaState(TableStateManager.java:177)
> org.apache.hadoop.hbase.master.TableStateManager.isTablePresent(TableStateManager.java:107)
> org.apache.hadoop.hbase.master.HMaster.getTableDescriptors(HMaster.java:3844)
> org.apache.hadoop.hbase.master.HMaster.listTableDescriptors(HMaster.java:3786){code}
> {code:java}
> this.controller = conn.rpcControllerFactory.newController();
> this.rpcControllerFactory =
> RpcControllerFactory.instantiate(conf);
> String rpcControllerFactoryClazz =
> configuration.get(CUSTOM_CONTROLLER_CONF_KEY,
> RpcControllerFactory.class.getName());
> set to ClientRpcControllerFactory.class.getName();{code}
>
> So for HBase 2.6 we use the previously set rpcControllerFactory, for HBase 3
> we use the CUSTOM_CONTROLLER_CONF_KEY which is ClientRpcControllerFactory and
> will create MetadataRpcController
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
