高建达 created HBASE-27983: --------------------------- Summary: The RSGroupAdminEndpoint is causing the hbase:meta region to be unable to come online. Key: HBASE-27983 URL: https://issues.apache.org/jira/browse/HBASE-27983 Project: HBase Issue Type: Bug Components: rsgroup Affects Versions: 2.1.1 Environment: *Hardware:*
Red Hat Enterprise Linux Server release 7.9 (Maipo) HDD 12 * 50G 16 cores *Software:* HBase Version : 2.1.1 Hadoop Version : hadoop-2.7.2 Zookeeper Version : 3.5.7 *Roles:* HBase is configured to use a master high availability (HA) mode with two masters and three regionservers. ||host||role|| |ysl102-qax.com|master regionserver| |ysl103-qax.com|master regionserver| |ysl104-qax.com|regionserver | Reporter: 高建达 Attachments: hbase-hbase-master-YSL104-QAX.COM.log When I use RSGroupAdminEndpoint and restart both master before restarting the regionserver, I encounter a syscall:getsockopt(..) issue that prevents the hbase:meta region from coming online, resulting in a service exception. {code:java} 2023-07-10 16:32:22,282 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-ysl104-qax.com,16000,1688977910162] rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread: Updating default servers. 2023-07-10 16:32:22,299 INFO [PEWorker-8] procedure.ServerCrashProcedure: Start pid=2, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=ysl102-qax.com,16020,1688977249460, splitWal=true, meta=false 2023-07-10 16:32:22,400 INFO [PEWorker-10] master.SplitLogManager: hdfs://HACluster/home/hbase/WALs/ysl102-qax.com,16020,1688977249460-splitting dir is empty, no logs to split. 2023-07-10 16:32:22,411 INFO [PEWorker-10] master.SplitLogManager: Finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://HACluster/home/hbase/WALs/ysl102-qax.com,16020,1688977249460-splitting] in 0ms 2023-07-10 16:32:22,521 INFO [PEWorker-10] procedure2.ProcedureExecutor: Finished pid=2, state=SUCCESS; ServerCrashProcedure server=ysl102-qax.com,16020,1688977249460, splitWal=true, meta=false in 326msec 2023-07-10 16:32:22,941 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node deleted, processing expiration [ysl104-qax.com,16020,1688977251592] 2023-07-10 16:32:22,941 INFO [RegionServerTracker-0] master.ServerManager: Processing expiration of ysl104-qax.com,16020,1688977251592 on ysl104-qax.com,16000,1688977910162 2023-07-10 16:32:23,069 INFO [PEWorker-12] procedure.ServerCrashProcedure: Start pid=3, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=ysl104-qax.com,16020,1688977251592, splitWal=true, meta=true 2023-07-10 16:32:23,126 INFO [PEWorker-12] master.SplitLogManager: hdfs://HACluster/home/hbase/WALs/ysl104-qax.com,16020,1688977251592-splitting dir is empty, no logs to split. 2023-07-10 16:32:23,135 INFO [PEWorker-12] master.SplitLogManager: Finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://HACluster/home/hbase/WALs/ysl104-qax.com,16020,1688977251592-splitting] in 0ms 2023-07-10 16:32:23,174 INFO [PEWorker-12] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740}] 2023-07-10 16:32:23,206 INFO [PEWorker-15] procedure.MasterProcedureScheduler: Took xlock for pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740 2023-07-10 16:32:23,325 INFO [PEWorker-15] assignment.AssignProcedure: Starting pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=ysl104-qax.com,16020,1688977251592; forceNewPlan=false, retain=true 2023-07-10 16:32:23,476 WARN [master/YSL104-QAX:16000] assignment.AssignmentManager: No servers available; cannot place 1 unassigned regions. 2023-07-10 16:32:24,477 WARN [master/YSL104-QAX:16000] assignment.AssignmentManager: No servers available; cannot place 1 unassigned regions. 2023-07-10 16:32:25,478 WARN [master/YSL104-QAX:16000] assignment.AssignmentManager: No servers available; cannot place 1 unassigned regions. 2023-07-10 16:32:26,479 WARN [master/YSL104-QAX:16000] assignment.AssignmentManager: No servers available; cannot place 1 unassigned regions. 2023-07-10 16:32:26,665 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-ysl104-qax.com,16000,1688977910162] client.RpcRetryingCallerImpl: Call exception, tries=6, retries=46, started=4175 ms ago, cancelled=false, msg=Call to YSL104-QAX.COM/10.59.12.104:16020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: YSL104-QAX.COM/xx.xx.xx.104:16020, details=row 'hbase:rsgroup' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=ysl104-qax.com,16020,1688977251592, seqNum=-1 2023-07-10 16:32:27,480 WARN [master/YSL104-QAX:16000] assignment.AssignmentManager: No servers available; cannot place 1 unassigned regions. 2023-07-10 16:32:28,481 WARN [master/YSL104-QAX:16000] assignment.AssignmentManager: No servers available; cannot place 1 unassigned regions. 2023-07-10 16:32:29,482 WARN [master/YSL104-QAX:16000] assignment.AssignmentManager: No servers available; cannot place 1 unassigned regions. 2023-07-10 16:32:30,483 WARN [master/YSL104-QAX:16000] assignment.AssignmentManager: No servers available; cannot place 1 unassigned regions. 2023-07-10 16:32:30,899 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-ysl104-qax.com,16000,1688977910162] client.RpcRetryingCallerImpl: Call exception, tries=7, retries=46, started=8409 ms ago, cancelled=false, msg=Connection closed, details=row 'hbase:rsgroup' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=ysl104-qax.com,16020,1688977251592, seqNum=-1 2023-07-10 16:32:31,025 INFO [RpcServer.default.FPBQ.Fifo.handler=198,queue=18,port=16000] master.ServerManager: Registering regionserver=ysl103-qax.com,16020,1688977946684 2023-07-10 16:32:31,064 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [ysl103-qax.com,16020,1688977946684] 2023-07-10 16:32:31,439 INFO [RpcServer.default.FPBQ.Fifo.handler=198,queue=18,port=16000] master.ServerManager: Registering regionserver=ysl102-qax.com,16020,1688977947399 2023-07-10 16:32:31,467 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [ysl102-qax.com,16020,1688977947399] 2023-07-10 16:32:32,934 INFO [RpcServer.default.FPBQ.Fifo.handler=198,queue=18,port=16000] master.ServerManager: Registering regionserver=ysl104-qax.com,16020,1688977948804 2023-07-10 16:32:32,965 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [ysl104-qax.com,16020,1688977948804] 2023-07-10 16:32:41,041 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-ysl104-qax.com,16000,1688977910162] client.RpcRetryingCallerImpl: ption: hbase:meta,,1 is not online on ysl104-qax.com,16020,1688977948804 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3316) at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3293) at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1431) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2449) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) , details=row 'hbase:rsgroup' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=ysl104-qax.com,16020,1688977251592, seqNum=-1 2023-07-10 16:32:51,123 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-ysl104-qax.com,16000,1688977910162] client.RpcRetryingCallerImpl: ption: hbase:meta,,1 is not online on ysl104-qax.com,16020,1688977948804 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3316) at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3293) at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1431) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2449) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) , details=row 'hbase:rsgroup' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=ysl104-qax.com,16020,1688977251592, seqNum=-1 2023-07-10 16:33:01,212 INFO [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$ServerEventsListenerThread-ysl104-qax.com,16000,1688977910162] client.RpcRetryingCallerImpl: ption: hbase:meta,,1 is not online on ysl104-qax.com,16020,1688977948804 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3316) at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3293) at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1431) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2449) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) , details=row 'hbase:rsgroup' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=ysl104-qax.com,16020,1688977251592, seqNum=-1 {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)