[
https://issues.apache.org/jira/browse/PHOENIX-7776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Richárd Antal updated PHOENIX-7776:
-----------------------------------
Description:
With HBase 3 there are a lot of tests not finishing because we ran out of
metadata handlers configure by METADATA_HANDLER_COUNT_ATTRIB.
Few tests where it is easily visible are:
ConcurrentGetTablesIT
instead of 2 , usually passes with 20 failes with 18
[https://github.com/richardantal/phoenix-1/blob/dff56a6946585944d1e34dd28de1b135d60ff425/phoenix-core/src/it/java/org/apache/phoenix/end2end/ConcurrentGetTablesIT.java#L74]
SingleMetadataHandlerIT
With 1 metadata handler and 1 server side handler it hangs after
{code:java}
[RpcServer.Metadata.Fifo.handler=0,queue=0,port=58485]
client.ConnectionUtils(526): Start fetching meta region location from
registry{code}
and there is no log line similar to
{code:java}
2026-02-24T12:13:20,440 DEBUG
[RpcServer.Metadata.Fifo.handler=0,queue=0,port=58706]
util.ReflectedFunctionCache(97): Populated cache for
org.apache.hadoop.hbase.filter.SingleColumnValueFilter in 0ms
2026-02-24T12:13:20,440 DEBUG
[RpcServer.Metadata.Fifo.handler=0,queue=0,port=58706]
util.ReflectedFunctionCache(97): Populated cache for
org.apache.hadoop.hbase.filter.BinaryComparator in 0ms {code}
With 2 metadata handlers and 0 server side handler it passes
With this change it also passes:
[https://github.com/richardantal/phoenix-1/blob/dff56a6946585944d1e34dd28de1b135d60ff425/phoenix-core/src/it/java/org/apache/phoenix/end2end/SingleMetadataHandlerIT.java#L57-L61]
I've looked into it and there is a difference how we check the hbase:meta table
For HBase 2.6
{code:java}
org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory.getController(ServerRpcControllerFactory.java:58)
org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory.newController(ServerRpcControllerFactory.java:41)
org.apache.hadoop.hbase.client.HTable.get(HTable.java:388)
this.rpcControllerFactory.newController()
if (rpcControllerFactory == null) {
this.rpcControllerFactory =
RpcControllerFactory.instantiate(configuration);
} else {
this.rpcControllerFactory = rpcControllerFactory;
}
{code}
For HBase 3
{code:java}
org.apache.hadoop.hbase.client.AsyncRpcRetryingCaller.<init>(AsyncRpcRetryingCaller.java:91)
this.controller = conn.rpcControllerFactory.newController();
this.rpcControllerFactory =
RpcControllerFactory.instantiate(conf);
String rpcControllerFactoryClazz =
configuration.get(CUSTOM_CONTROLLER_CONF_KEY,
RpcControllerFactory.class.getName());
set to ClientRpcControllerFactory.class.getName();{code}
So for HBase 2.6 we use the previously set rpcControllerFactory, for HBase 3 we
use the CUSTOM_CONTROLLER_CONF_KEY which is ClientRpcControllerFactory and will
create MetadataRpcController
was:
With HBase 3 there are a lot of tests not finishing because we ran out of
metadata handlers configure by METADATA_HANDLER_COUNT_ATTRIB.
Few tests where it is easily visible are:
ConcurrentGetTablesIT
instead of 2 , usually passes with 20 failes with 18
[https://github.com/richardantal/phoenix-1/blob/dff56a6946585944d1e34dd28de1b135d60ff425/phoenix-core/src/it/java/org/apache/phoenix/end2end/ConcurrentGetTablesIT.java#L74]
SingleMetadataHandlerIT
With 1 metadata handler and 1 server side handler it hangs after
{code:java}
[RpcServer.Metadata.Fifo.handler=0,queue=0,port=58485]
client.ConnectionUtils(526): Start fetching meta region location from
registry{code}
and there is no log line similar to
{code:java}
2026-02-24T12:13:20,440 DEBUG
[RpcServer.Metadata.Fifo.handler=0,queue=0,port=58706]
util.ReflectedFunctionCache(97): Populated cache for
org.apache.hadoop.hbase.filter.SingleColumnValueFilter in 0ms
2026-02-24T12:13:20,440 DEBUG
[RpcServer.Metadata.Fifo.handler=0,queue=0,port=58706]
util.ReflectedFunctionCache(97): Populated cache for
org.apache.hadoop.hbase.filter.BinaryComparator in 0ms {code}
With 2 metadata handlers and 0 server side handler it passes
With this change it also passes:
[https://github.com/richardantal/phoenix-1/blob/dff56a6946585944d1e34dd28de1b135d60ff425/phoenix-core/src/it/java/org/apache/phoenix/end2end/SingleMetadataHandlerIT.java#L57-L61]
I've looked into it and there is a difference how we check the hbase:meta table
For HBase 2.6
{code:java}
org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory.getController(ServerRpcControllerFactory.java:58)
org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory.newController(ServerRpcControllerFactory.java:41)
org.apache.hadoop.hbase.client.HTable.get(HTable.java:388)
this.rpcControllerFactory.newController()
if (rpcControllerFactory == null) {
this.rpcControllerFactory =
RpcControllerFactory.instantiate(configuration);
} else {
this.rpcControllerFactory = rpcControllerFactory;
}
{code}
For HBase 3
{code:java}
org.apache.hadoop.hbase.ipc.controller.ClientRpcControllerFactory.getController(ClientRpcControllerFactory.java:40)
org.apache.phoenix.compat.hbase.CompatRPCControllerFactory.newController(CompatRPCControllerFactory.java:36)
org.apache.hadoop.hbase.client.AsyncRpcRetryingCaller.<init>(AsyncRpcRetryingCaller.java:91)
this.controller = conn.rpcControllerFactory.newController();
this.rpcControllerFactory = RpcControllerFactory.instantiate(conf);
String rpcControllerFactoryClazz =
configuration.get(CUSTOM_CONTROLLER_CONF_KEY,
RpcControllerFactory.class.getName()); set to
ClientRpcControllerFactory.class.getName();{code}
So for HBase 2.6 we use the previously set rpcControllerFactory, for HBase 3 we
use the CUSTOM_CONTROLLER_CONF_KEY which is ClientRpcControllerFactory and will
create MetadataRpcController
> Fix hanging tests because of Metadata handlerss
> -----------------------------------------------
>
> Key: PHOENIX-7776
> URL: https://issues.apache.org/jira/browse/PHOENIX-7776
> Project: Phoenix
> Issue Type: Task
> Reporter: Richárd Antal
> Priority: Major
>
> With HBase 3 there are a lot of tests not finishing because we ran out of
> metadata handlers configure by METADATA_HANDLER_COUNT_ATTRIB.
> Few tests where it is easily visible are:
> ConcurrentGetTablesIT
> instead of 2 , usually passes with 20 failes with 18
> [https://github.com/richardantal/phoenix-1/blob/dff56a6946585944d1e34dd28de1b135d60ff425/phoenix-core/src/it/java/org/apache/phoenix/end2end/ConcurrentGetTablesIT.java#L74]
> SingleMetadataHandlerIT
> With 1 metadata handler and 1 server side handler it hangs after
> {code:java}
> [RpcServer.Metadata.Fifo.handler=0,queue=0,port=58485]
> client.ConnectionUtils(526): Start fetching meta region location from
> registry{code}
> and there is no log line similar to
> {code:java}
> 2026-02-24T12:13:20,440 DEBUG
> [RpcServer.Metadata.Fifo.handler=0,queue=0,port=58706]
> util.ReflectedFunctionCache(97): Populated cache for
> org.apache.hadoop.hbase.filter.SingleColumnValueFilter in 0ms
> 2026-02-24T12:13:20,440 DEBUG
> [RpcServer.Metadata.Fifo.handler=0,queue=0,port=58706]
> util.ReflectedFunctionCache(97): Populated cache for
> org.apache.hadoop.hbase.filter.BinaryComparator in 0ms {code}
> With 2 metadata handlers and 0 server side handler it passes
> With this change it also passes:
> [https://github.com/richardantal/phoenix-1/blob/dff56a6946585944d1e34dd28de1b135d60ff425/phoenix-core/src/it/java/org/apache/phoenix/end2end/SingleMetadataHandlerIT.java#L57-L61]
> I've looked into it and there is a difference how we check the hbase:meta
> table
> For HBase 2.6
>
> {code:java}
> org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory.getController(ServerRpcControllerFactory.java:58)
> org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory.newController(ServerRpcControllerFactory.java:41)
> org.apache.hadoop.hbase.client.HTable.get(HTable.java:388)
> this.rpcControllerFactory.newController()
> if (rpcControllerFactory == null) {
> this.rpcControllerFactory =
> RpcControllerFactory.instantiate(configuration);
> } else {
> this.rpcControllerFactory = rpcControllerFactory;
> }
> {code}
>
> For HBase 3
> {code:java}
> org.apache.hadoop.hbase.client.AsyncRpcRetryingCaller.<init>(AsyncRpcRetryingCaller.java:91)
> this.controller = conn.rpcControllerFactory.newController();
> this.rpcControllerFactory =
> RpcControllerFactory.instantiate(conf);
> String rpcControllerFactoryClazz =
> configuration.get(CUSTOM_CONTROLLER_CONF_KEY,
> RpcControllerFactory.class.getName());
> set to ClientRpcControllerFactory.class.getName();{code}
>
> So for HBase 2.6 we use the previously set rpcControllerFactory, for HBase 3
> we use the CUSTOM_CONTROLLER_CONF_KEY which is ClientRpcControllerFactory and
> will create MetadataRpcController
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
