Ankit Singhal created HBASE-25166:
-------------------------------------
Summary: MobFileCompactionChore is closing the master's shared
cluster connection
Key: HBASE-25166
URL: https://issues.apache.org/jira/browse/HBASE-25166
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 3.0.0-alpha-1
Reporter: Ankit Singhal
Assignee: Ankit Singhal
Code which is doing so in MobFileCompactionChore
{code:java}
try (Connection conn = master.getConnection();
Admin admin = conn.getAdmin();) { {code}
As master uses this connection to read the meta or other system tables, so none
of the meta operations through master will work.
Symptoms in master logs:-
{code:java}
s, events=841, succcessCount=123, totalEvents=12824192,
totalSuccessCount=1891300
2020-10-05 16:34:25,062 INFO
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor: Unable to
get remote Address
2020-10-05 16:34:25,062 ERROR
org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore: Failed to
normalize regions.
java.io.IOException: connection is closed
at
org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:241)
at
org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:797)
at
org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:768)
at
org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:727)
at
org.apache.hadoop.hbase.MetaTableAccessor.fullScanTables(MetaTableAccessor.java:215)
at
org.apache.hadoop.hbase.master.TableStateManager.getTablesInStates(TableStateManager.java:189)
at
org.apache.hadoop.hbase.master.HMaster.normalizeRegions(HMaster.java:1821)
at
org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore.chore(RegionNormalizerChore.java:48)
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:188)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) {code}
Symptoms at the client:-
{code:java}
RpcRetryingCaller{globalStartTime=1602099132430, pause=100, maxAttempts=11},
java.io.IOException: java.io.IOException: connection is closed
at
org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:241)
at
org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1116)
at
org.apache.hadoop.hbase.master.TableStateManager.readMetaState(TableStateManager.java:258)
at
org.apache.hadoop.hbase.master.TableStateManager.isTablePresent(TableStateManager.java:175)
at
org.apache.hadoop.hbase.master.HMaster.getTableDescriptors(HMaster.java:3277)
at
org.apache.hadoop.hbase.master.HMaster.listTableDescriptors(HMaster.java:3221)
at
org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1064)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:418)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
