[
https://issues.apache.org/jira/browse/HBASE-25954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17355507#comment-17355507
]
Baiqiang Zhao commented on HBASE-25954:
---------------------------------------
HBASE-22740 was meant to solve this problem, but branch-2 did not solve it, it
has some conflicts. If the regionservers in your rsgroup do not change
frequently, you can try the designatedfile or excludefile parameters in
graceful_stop.sh.
> When graceful_stop a regionserver, RegionMover always retry to move the
> regions to other rsgoup's regionservers
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-25954
> URL: https://issues.apache.org/jira/browse/HBASE-25954
> Project: HBase
> Issue Type: Bug
> Reporter: Zezhen Jia
> Priority: Critical
>
> [had...@bigdata-hbase-master003.prod.com bin]$ ./graceful_stop.sh
> bigdata-hbase-rs105.prod.com 2021-05-28T11:05:06 Disabling load balancer
> SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in
> [jar:file:/home/hadoop/cloud/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/hadoop/cloud/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an
> explanation. SLF4J: Actual binding is of type
> [org.slf4j.impl.Log4jLoggerFactory] 2021-05-28T11:05:12 Previous balancer
> state was true 2021-05-28T11:05:12 Unloading bigdata-hbase-rs105.prod.com
> region(s) SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found
> binding in
> [jar:file:/home/hadoop/cloud/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/hadoop/cloud/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an
> explanation. SLF4J: Actual binding is of type
> [org.slf4j.impl.Log4jLoggerFactory] 2021-05-28 11:05:13,990 INFO
> [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c]
> zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f,
> built on 03/23/2017 10:13 GMT 2021-05-28 11:05:13,991 INFO
> [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c]
> zookeeper.ZooKeeper: Client
> environment:host.name=bigdata-hbase-master003.prod.com 2021-05-28
> 11:05:13,991 INFO
> [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c]
> zookeeper.ZooKeeper: Client environment:java.version=1.8.0_241 2021-05-28
> 11:05:13,991 INFO
> [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c]
> zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
> 2021-05-28 11:05:13,991 INFO
> [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c]
> zookeeper.ZooKeeper: Client
> environment:java.home=/home/hadoop/cloud/jdk1.8.0_241/jre 2021-05-28
> 11:05:13,991 INFO
> [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c]
> zookeeper.ZooKeeper:
> adoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/cloud/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.9.2.jar:/home/hadoop/cloud/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.9.2-tests.jar:/home/hadoop/cloud/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar:/home/hadoop/cloud/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.9.2.jar:/home/hadoop/cloud/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.9.2.jar:/home/hadoop/cloud/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.9.2.jar:/home/hadoop/cloud/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.9.2.jar:/home/hadoop/cloud/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.9.2.jar:/home/hadoop/cloud/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.9.2.jar:/home/hadoop/cloud/hadoop/contrib/capacity-scheduler/*.jar:/home/hadoop/cloud/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar
> 2021-05-28 11:05:13,991 INFO
> [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c]
> zookeeper.ZooKeeper: Client
> environment:java.library.path=/home/hadoop/cloud/hadoop/lib/native 2021-05-28
> 11:05:13,991 INFO
> [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c]
> zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2021-05-28
> 11:05:13,991 INFO
> [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c]
> zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 2021-05-28
> 11:05:13,991 INFO
> [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c]
> zookeeper.ZooKeeper: Client environment:os.name=Linux 2021-05-28
> 11:05:13,991 INFO
> [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c]
> zookeeper.ZooKeeper: Client environment:os.arch=amd64 2021-05-28
> 11:05:13,991 INFO
> [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c]
> zookeeper.ZooKeeper: Client environment:os.version=2.6.32-696.3.1.el6.x86_64
> 2021-05-28 11:05:13,991 INFO
> [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c]
> zookeeper.ZooKeeper: Client environment:user.name=hadoop 2021-05-28
> 11:05:13,991 INFO
> [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c]
> zookeeper.ZooKeeper: Client environment:user.home=/home/hadoop 2021-05-28
> 11:05:13,992 INFO
> [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c]
> zookeeper.ZooKeeper: Client
> environment:user.dir=/home/hadoop/cloud/hbase/bin 2021-05-28 11:05:13,993
> INFO
> [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c]
> zookeeper.ZooKeeper: Initiating client connection,
> connectString=bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181
> sessionTimeout=30000
> watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$14/668288113@5d84a9ed
> 2021-05-28 11:05:14,138 INFO
> [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c-SendThread(bigdata-hbase-rs101.prod.com:2181)]
> zookeeper.Login: Client successfully logged in. 2021-05-28 11:05:14,140 INFO
> [Thread-4] zookeeper.Login: TGT refresh thread started. 2021-05-28
> 11:05:14,142 INFO [Thread-4] zookeeper.Login: TGT valid starting at: Fri May
> 28 11:05:14 CST 2021 2021-05-28 11:05:14,142 INFO [Thread-4] zookeeper.Login:
> TGT expires: Fri May 28 23:05:14 CST 2021 2021-05-28 11:05:14,143 INFO
> [Thread-4] zookeeper.Login: TGT refresh sleeping until: Fri May 28 21:07:54
> CST 2021 2021-05-28 11:05:14,144 INFO
> [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c-SendThread(bigdata-hbase-rs101.prod.com:2181)]
> client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
> 2021-05-28 11:05:14,150 INFO
> [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c-SendThread(bigdata-hbase-rs101.prod.com:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server
> bigdata-hbase-rs101.prod.com/10.87.103.134:2181. Will attempt to
> SASL-authenticate using Login Context section 'Client' 2021-05-28
> 11:05:14,154 INFO
> [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c-SendThread(bigdata-hbase-rs101.prod.com:2181)]
> zookeeper.ClientCnxn: Socket connection established to
> bigdata-hbase-rs101.prod.com/10.87.103.134:2181, initiating session
> 2021-05-28 11:05:14,158 INFO
> [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c-SendThread(bigdata-hbase-rs101.prod.com:2181)]
> zookeeper.ClientCnxn: Session establishment complete on server
> bigdata-hbase-rs101.prod.com/10.87.103.134:2181, sessionid =
> 0x371ba929aec16a2, negotiated timeout = 30000 2021-05-28 11:05:14,754 INFO
> [pool-2-thread-1] util.RegionMover: Moving 16 regions from
> bigdata-hbase-rs105.prod.com to 4 servers using 1 threads .Ack Mode:true
> 2021-05-28 11:05:20,479 INFO [pool-3-thread-1] util.RegionMover: Moving
> region:0e75b85e4970faf3ee7d83ba5edfbf7a from
> bigdata-hbase-rs105.prod.com,16020,1616402972076 to
> bigdata-hbase-rs101.prod.com,16020,1615279448828
> 2021-05-28 11:06:14,775 INFO [pool-3-thread-1] util.RegionMover: Retry 1 of
> maximum 5 2021-05-28 11:06:14,786 INFO
> [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c]
> zookeeper.ZooKeeper: Session: 0x371ba929aec16a2 closed 2021-05-28
> 11:06:14,786 INFO
> [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c-EventThread]
> zookeeper.ClientCnxn: EventThread shut down for session: 0x371ba929aec16a2
> 2021-05-28 11:06:18,880 INFO [pool-3-thread-1] util.RegionMover: Retry 2 of
> maximum 5 2021-05-28 11:06:19,983 INFO [pool-3-thread-1] util.RegionMover:
> Retry 3 of maximum 5 2021-05-28 11:06:21,086 INFO [pool-3-thread-1]
> util.RegionMover: Retry 4 of maximum 5 2021-05-28 11:06:22,190 ERROR
> [pool-3-thread-1] util.RegionMover: Region: test:t2,080,
> !1r^,1609991498659.0e75b85e4970faf3ee7d83ba5edfbf7a. stuck on
> bigdata-hbase-rs105.prod.com,16020,1616402972076,newServer=bigdata-hbase-rs101.prod.com,16020,1615279448828
> 2021-05-28 11:06:22,210 INFO [pool-3-thread-1] util.RegionMover: Moving
> region:cad8bd3f09cb02b6be55acfa3fe81509 from
> bigdata-hbase-rs105.prod.com,16020,1616402972076 to
> bigdata-hbase-rs102.prod.com,16020,1615279437177
> 2021-05-28 11:07:22,265 INFO [pool-3-thread-1] util.RegionMover: Retry 1 of
> maximum 5 2021-05-28 11:07:23,369 INFO [pool-3-thread-1] util.RegionMover:
> Retry 2 of maximum 5 2021-05-28 11:07:24,472 INFO [pool-3-thread-1]
> util.RegionMover: Retry 3 of maximum 5 2021-05-28 11:07:25,576 INFO
> [pool-3-thread-1] util.RegionMover: Retry 4 of maximum 5 2021-05-28
> 11:07:26,679 ERROR [pool-3-thread-1] util.RegionMover: Region: test:t2,0c0,
> 4B.J,1609991498659.cad8bd3f09cb02b6be55acfa3fe81509. stuck on
> bigdata-hbase-rs105.prod.com,16020,1616402972076,newServer=bigdata-hbase-rs102.prod.com,16020,1615279437177
> 2021-05-28 11:07:26,699 INFO [pool-3-thread-1] util.RegionMover: Moving
> region:175b9149fe5c2715832fe8ebe2273e3e from
> bigdata-hbase-rs105.prod.com,16020,1616402972076 to
> bigdata-hbase-rs103.prod.com,16020,1615279437032 2021-05-28 11:07:27,868 INFO
> [pool-3-thread-1] util.RegionMover: Moved Region test:t2,100,
> "la,1608747109269.175b9149fe5c2715832fe8ebe2273e3e. cost:1.188 2021-05-28
> 11:07:27,890 INFO [pool-3-thread-1] util.RegionMover: Moving
> region:8e58792c240f9dd6879fa610de648978 from
> bigdata-hbase-rs105.prod.com,16020,1616402972076 to
> bigdata-hbase-rs104.prod.com,16020,1615279437314 2021-05-28 11:07:29,042 INFO
> [pool-3-thread-1] util.RegionMover: Moved Region test:t2,180,
> &C9#,1608747109269.8e58792c240f9dd6879fa610de648978. cost:1.174 2021-05-28
> 11:07:29,061 INFO [pool-3-thread-1] util.RegionMover: Moving
> region:b135ab7358a5375eabec1ba9bd6abea2 from
> bigdata-hbase-rs105.prod.com,16020,1616402972076 to
> bigdata-hbase-rs101.prod.com,16020,1615279448828 2021-05-28 11:08:29,144 INFO
> [pool-3-thread-1] util.RegionMover: Retry 1 of maximum 5 2021-05-28
> 11:08:30,248 INFO [pool-3-thread-1] util.RegionMover: Retry 2 of maximum 5
> 2021-05-28 11:08:31,351 INFO [pool-3-thread-1] util.RegionMover: Retry 3 of
> maximum 5 2021-05-28 11:08:32,454 INFO [pool-3-thread-1] util.RegionMover:
> Retry 4 of maximum 5 2021-05-28 11:08:33,556 ERROR [pool-3-thread-1]
> util.RegionMover: Region:
> test:t2,280,101866,1608761306479.b135ab7358a5375eabec1ba9bd6abea2. stuck on
> bigdata-hbase-rs105.prod.com,16020,1616402972076,newServer=bigdata-hbase-rs101.prod.com,16020,1615279448828
> 2021-05-28 11:08:33,576 INFO [pool-3-thread-1] util.RegionMover: Moving
> region:9ab60065e4d739b4b1205baaeb4ab112 from
> bigdata-hbase-rs105.prod.com,16020,1616402972076 to
> bigdata-hbase-rs102.prod.com,16020,1615279437177 2021-05-28 11:09:33,626 INFO
> [pool-3-thread-1] util.RegionMover: Retry 1 of maximum 5 2021-05-28
> 11:09:34,729 INFO [pool-3-thread-1] util.RegionMover: Retry 2 of maximum 5
> 2021-05-28 11:09:35,833 INFO [pool-3-thread-1] util.RegionMover: Retry 3 of
> maximum 5 2021-05-28 11:09:36,936 INFO [pool-3-thread-1] util.RegionMover:
> Retry 4 of maximum 5 2021-05-28 11:09:38,038 ERROR [pool-3-thread-1]
> util.RegionMover: Region:
> test:t2,380,7408,1608742520235.9ab60065e4d739b4b1205baaeb4ab112. stuck on
> bigdata-hbase-rs105.prod.com,16020,1616402972076,newServer=bigdata-hbase-rs102.prod.com,16020,1615279437177
> 2021-05-28 11:09:38,052 INFO [pool-3-thread-1] util.RegionMover: Moving
> region:36d483f9537cbcb066c535d5c97500b7 from
> bigdata-hbase-rs105.prod.com,16020,1616402972076 to
> bigdata-hbase-rs103.prod.com,16020,1615279437032
> 2021-05-28 11:10:38,056 INFO [pool-3-thread-1] util.RegionMover: Retry 1 of
> maximum 5 2021-05-28 11:10:39,159 INFO [pool-3-thread-1] util.RegionMover:
> Retry 2 of maximum 5 2021-05-28 11:10:40,261 INFO [pool-3-thread-1]
> util.RegionMover: Retry 3 of maximum 5 2021-05-28 11:10:41,364 INFO
> [pool-3-thread-1] util.RegionMover: Retry 4 of maximum 5 2021-05-28
> 11:10:42,467 ERROR [pool-3-thread-1] util.RegionMover: Region:
> test:t2,480,7627,1608755031336.36d483f9537cbcb066c535d5c97500b7. stuck on
> bigdata-hbase-rs105.prod.com,16020,1616402972076,newServer=bigdata-hbase-rs103.prod.com,16020,1615279437032
> 2021-05-28 11:10:42,491 INFO [pool-3-thread-1] util.RegionMover: Moving
> region:4fb21d1bcdd1230351828b7c6e779e08 from
> bigdata-hbase-rs105.prod.com,16020,1616402972076 to
> bigdata-hbase-rs104.prod.com,16020,1615279437314 2021-05-28 11:10:43,629 INFO
> [pool-3-thread-1] util.RegionMover: Moved Region
> test:t2,60000000,1608752443610.4fb21d1bcdd1230351828b7c6e779e08. cost:1.161
> 2021-05-28 11:10:43,659 INFO [pool-3-thread-1] util.RegionMover: Moving
> region:3286929984e362752b03dd4f1c723b78 from
> bigdata-hbase-rs105.prod.com,16020,1616402972076 to
> bigdata-hbase-rs101.prod.com,16020,1615279448828
> 2021-05-28 11:11:43,652 INFO [pool-3-thread-1] util.RegionMover: Retry 1 of
> maximum 5 2021-05-28 11:11:44,755 INFO [pool-3-thread-1] util.RegionMover:
> Retry 2 of maximum 5 2021-05-28 11:11:45,858 INFO [pool-3-thread-1]
> util.RegionMover: Retry 3 of maximum 5 2021-05-28 11:11:46,961 INFO
> [pool-3-thread-1] util.RegionMover: Retry 4 of maximum 5 2021-05-28
> 11:11:48,063 ERROR [pool-3-thread-1] util.RegionMover: Region:
> test:t2,700,73323,1608755402890.3286929984e362752b03dd4f1c723b78. stuck on
> bigdata-hbase-rs105.prod.com,16020,1616402972076,newServer=bigdata-hbase-rs101.prod.com,16020,1615279448828
> 2021-05-28 11:11:48,089 INFO [pool-3-thread-1] util.RegionMover: Moving
> region:1626df981007111128002bb7008ebe07 from
> bigdata-hbase-rs105.prod.com,16020,1616402972076 to
> bigdata-hbase-rs102.prod.com,16020,1615279437177 2021-05-28 11:11:49,209 INFO
> [pool-3-thread-1] util.RegionMover: Moved Region
> test:t2,80000000,1608747946269.1626df981007111128002bb7008ebe07. cost:1.146
> 2021-05-28 11:11:49,234 INFO [pool-3-thread-1] util.RegionMover: Moving
> region:b9934989ffd9abb8b2484208b2a4860c from
> bigdata-hbase-rs105.prod.com,16020,1616402972076 to
> bigdata-hbase-rs103.prod.com,16020,1615279437032 2021-05-28 11:12:49,215 INFO
> [pool-3-thread-1] util.RegionMover: Retry 1 of maximum 5 2021-05-28
> 11:12:50,318 INFO [pool-3-thread-1] util.RegionMover: Retry 2 of maximum 5
> 2021-05-28 11:12:51,421 INFO [pool-3-thread-1] util.RegionMover: Retry 3 of
> maximum 5 2021-05-28 11:12:52,523 INFO [pool-3-thread-1] util.RegionMover:
> Retry 4 of maximum 5 2021-05-28 11:12:53,626 ERROR [pool-3-thread-1]
> util.RegionMover: Region:
> test:t2,900,7405,1608752559891.b9934989ffd9abb8b2484208b2a4860c. stuck on
> bigdata-hbase-rs105.prod.com,16020,1616402972076,newServer=bigdata-hbase-rs103.prod.com,16020,1615279437032
> 2021-05-28 11:12:53,657 INFO [pool-3-thread-1] util.RegionMover: Moving
> region:2cfb6d72b252e02b6651741bce8af0a9 from
> bigdata-hbase-rs105.prod.com,16020,1616402972076 to
> bigdata-hbase-rs104.prod.com,16020,1615279437314 2021-05-28 11:12:54,789 INFO
> [pool-3-thread-1] util.RegionMover: Moved Region
> test:t2,980,5003,1608752559891.2cfb6d72b252e02b6651741bce8af0a9. cost:1.163
> 2021-05-28 11:12:54,805 INFO [pool-3-thread-1] util.RegionMover: Moving
> region:027a58629de39ed42fd534364f239bc1 from
> bigdata-hbase-rs105.prod.com,16020,1616402972076 to
> bigdata-hbase-rs101.prod.com,16020,1615279448828 2021-05-28 11:12:55,918 INFO
> [pool-3-thread-1] util.RegionMover: Moved Region
> test:t2,a0000000,1608748813503.027a58629de39ed42fd534364f239bc1. cost:1.129
> 2021-05-28 11:12:55,941 INFO [pool-3-thread-1] util.RegionMover: Moving
> region:2f40e2ea776ad5ccc63dedc112f780b3 from
> bigdata-hbase-rs105.prod.com,16020,1616402972076 to
> bigdata-hbase-rs102.prod.com,16020,1615279437177
> 2021-05-28 11:13:55,918 INFO [pool-3-thread-1] util.RegionMover: Retry 1 of
> maximum 5 2021-05-28 11:13:57,022 INFO [pool-3-thread-1] util.RegionMover:
> Retry 2 of maximum 5 2021-05-28 11:13:58,124 INFO [pool-3-thread-1]
> util.RegionMover: Retry 3 of maximum 5 2021-05-28 11:13:59,227 INFO
> [pool-3-thread-1] util.RegionMover: Retry 4 of maximum 5 ^[2021-05-28
> 11:14:00,330 ERROR [pool-3-thread-1] util.RegionMover: Region:
> test:t2,c0000000,1608749249432.2f40e2ea776ad5ccc63dedc112f780b3. stuck on
> bigdata-hbase-rs105.prod.com,16020,1616402972076,newServer=bigdata-hbase-rs102.prod.com,16020,1615279437177
> 2021-05-28 11:14:00,376 INFO [pool-3-thread-1] util.RegionMover: Moving
> region:bebf648e765b7369ff2a86c6ca290828 from
> bigdata-hbase-rs105.prod.com,16020,1616402972076 to
> bigdata-hbase-rs103.prod.com,16020,1615279437032 2021-05-28 11:15:00,409 INFO
> [pool-3-thread-1] util.RegionMover: Retry 1 of maximum 5 2021-05-28
> 11:15:01,512 INFO [pool-3-thread-1] util.RegionMover: Retry 2 of maximum 5
> 2021-05-28 11:15:02,615 INFO [pool-3-thread-1] util.RegionMover: Retry 3 of
> maximum 5 2021-05-28 11:15:03,718 INFO [pool-3-thread-1] util.RegionMover:
> Retry 4 of maximum 5 2021-05-28 11:15:04,821 ERROR [pool-3-thread-1]
> util.RegionMover: Region: test:t2,e80,
> _R@_,1608746950666.bebf648e765b7369ff2a86c6ca290828. stuck on
> bigdata-hbase-rs105.prod.com,16020,1616402972076,newServer=bigdata-hbase-rs103.prod.com,16020,1615279437032
> 2021-05-28 11:15:04,844 INFO [pool-3-thread-1] util.RegionMover: Moving
> region:5beda27c11cef510a70af7f4609c84d3 from
> bigdata-hbase-rs105.prod.com,16020,1616402972076 to
> bigdata-hbase-rs104.prod.com,16020,1615279437314 2021-05-28 11:15:05,994 INFO
> [pool-3-thread-1] util.RegionMover: Moved Region test:t2,f80,
> !<&2,1608745215226.5beda27c11cef510a70af7f4609c84d3. cost:1.172 2021-05-28
> 11:15:05,994 ERROR [pool-2-thread-1] util.RegionMover: Was Not able to move
> region....Exiting Now 2021-05-28 11:15:05,995 ERROR [pool-2-thread-1]
> util.RegionMover: Error while unloading regions java.lang.Exception: Could
> not move region Exception at
> org.apache.hadoop.hbase.util.RegionMover.waitMoveTasksToFinish(RegionMover.java:541)
> at
> org.apache.hadoop.hbase.util.RegionMover.unloadRegions(RegionMover.java:497)
> at
> org.apache.hadoop.hbase.util.RegionMover.lambda$unload$1(RegionMover.java:449)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) 2021-05-28 11:15:06,003 INFO [main]
> client.ConnectionImplementation: Closing master protocol: MasterService
> 2021-05-28T11:15:06 Unloaded bigdata-hbase-rs105.prod.com region(s)
> 2021-05-28T11:15:06 Stopping regionserver on bigdata-hbase-rs105.prod.com
> bigdata-hbase-rs105.prod.com: running regionserver, logging to
> /home/logs/hbase/hbase-hadoop-regionserver-bigdata-hbase-rs105.prod.com.out
> bigdata-hbase-rs105.prod.com: stopping regionserver.... 2021-05-28T11:15:10
> Restoring balancer state to true
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
