hi
????????????RS????????????????????????????????????????????????????????????????????????????????????????????patch,???????? ------------------ ???????? ------------------ ??????: "user-zh" <palomino...@gmail.com>; ????????: 2021??1??5??(??????) ????4:48 ??????: "user-zh"<user-zh@hbase.apache.org>; ????: Re: ????HBase????????HBase2.1.0 ???????????????????????? ????????????????????????bug????????????????????????????last flushed sequnce id????????????????regionserver???????? ???? <xiaozhang0...@163.com> ??2021??1??5?????? ????4:40?????? > > ????????????????????????HLog??????????????????region??????????region??????????split??????????????HLog????????????????????WAL??????????????????????????replication???????? > > > ????11??21:02.736?? INFO AbstractFSWAL Too many WALs; count=16512, max=256; > forcing flush of 2 regions(s): ce7c15393f6021611c12eb8f7739c55a, > a9e65042043cf6c122785aee3dd52d98????11??21:02.736?? WARN LogRoller Failed to > schedule flush of ce7c15393f6021611c12eb8f7739c55a,region=null, > requester=null > > > > > | | > ???? > xiaozhang0...@163.com > ?????????????????? > | > ?????????????????????? > ??2021??1??4?? 22:10??sudo rm -rf /*<2326130...@qq.com> ?????? > hi > &nbsp; &nbsp; ????????????HBase??????1.2??????????replication??????????????????????????????HBase2.x > ??replication????????????????????????????????????????????????????2.1????????????????????????????????replication???????? > &nbsp; &nbsp; > ??????2.1??????replication?????????????????????????????????????????????????????????????????????????????????????????????????????????????? > > > > > ------------------&nbsp;????????&nbsp;------------------ > ??????: > "user-zh" > < > palomino...@gmail.com&gt;; > ????????:&nbsp;2021??1??4??(??????) ????10:01 > ??????:&nbsp;"user-zh"<user-zh@hbase.apache.org&gt;; > > ????:&nbsp;Re: ????HBase????????HBase2.1.0 ???????????????????????? > > > > ?????????????????????? > > ????????????serial replication????????????????????????????????????bug???????????????????????????????????????????? > > ????(Duo Zhang) <palomino...@gmail.com&gt; ??2021??1??4?????? ????9:58?????? > > &gt; shell???? > &gt; > &gt; list_peers > &gt; > &gt; sudo rm -rf /* <2326130...@qq.com&gt; ??2021??1??4?????? ????9:33?????? > &gt; > &gt;&gt; hi > &gt;&gt;&nbsp; ????????????????????????Replication??ERROR??????replication > peer????????????????????????? > &gt;&gt; ???????????????????????? > &gt;&gt; > &gt;&gt; hbase(main):002:0&gt; list_peers > &gt;&gt;&nbsp; PEER_ID CLUSTER_KEY ENDPOINT_CLASSNAME STATE REPLICATE_ALL > NAMESPACES > &gt;&gt; TABLE_CFS BANDWIDTH SERIAL > &gt;&gt;&nbsp; 1 10.2.5.206:2181,10.2.5.208:2181,10.2.5.209:2181:/hbase&nbsp; > ENABLED true > &gt;&gt;&nbsp; 0 true > &gt;&gt; 1 row(s) > &gt;&gt; Took 0.0463 seconds > &gt;&gt; > &gt;&gt; > &gt;&gt; =&gt; #<Java::JavaUtil::ArrayList:0x79d49790&gt; > &gt;&gt; > &gt;&gt; hbase(main):003:0&gt; status 'replication' > &gt;&gt; version 2.1.0-cdh6.3.2 > &gt;&gt; 5 live servers > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; > centos-bigdata-datanode-10-2-5-197.intsig.internal: > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SOURCE: PeerID=1, > AgeOfLastShippedOp=0, SizeOfLogQueue=11640, > &gt;&gt; TimeStampsOfLastShippedOp=Mon Jan 04 21:29:44 CST 2021, > Replication > &gt;&gt; Lag=17631 > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SINK&nbsp; : > AgeOfLastAppliedOp=90, TimeStampsOfLastAppliedOp=Mon Jan > &gt;&gt; 04 21:30:01 CST 2021 > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; > centos-bigdata-datanode-10-2-5-198.intsig.internal: > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SOURCE: PeerID=1, > AgeOfLastShippedOp=40642518, > &gt;&gt; SizeOfLogQueue=36125, TimeStampsOfLastShippedOp=Mon Jan 04 > 21:29:49 CST > &gt;&gt; 2021, Replication Lag=40642518 > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SINK&nbsp; : > AgeOfLastAppliedOp=42, TimeStampsOfLastAppliedOp=Mon Jan > &gt;&gt; 04 21:30:01 CST 2021 > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; > centos-bigdata-datanode-10-2-5-199.intsig.internal: > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SOURCE: PeerID=1, > AgeOfLastShippedOp=40761400, > &gt;&gt; SizeOfLogQueue=15745, TimeStampsOfLastShippedOp=Mon Jan 04 > 21:29:55 CST > &gt;&gt; 2021, Replication Lag=40761400 > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SINK&nbsp; : > AgeOfLastAppliedOp=86, TimeStampsOfLastAppliedOp=Mon Jan > &gt;&gt; 04 21:30:01 CST 2021 > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; > centos-bigdata-datanode-10-2-5-200.intsig.internal: > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SOURCE: PeerID=1, > AgeOfLastShippedOp=4, SizeOfLogQueue=86857, > &gt;&gt; TimeStampsOfLastShippedOp=Mon Jan 04 21:29:59 CST 2021, > Replication Lag=2979 > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SINK&nbsp; : > AgeOfLastAppliedOp=85, TimeStampsOfLastAppliedOp=Mon Jan > &gt;&gt; 04 21:30:01 CST 2021 > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; > centos-bigdata-datanode-10-2-5-205.intsig.internal: > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SOURCE: PeerID=1, > AgeOfLastShippedOp=40698192, > &gt;&gt; SizeOfLogQueue=15046, TimeStampsOfLastShippedOp=Mon Jan 04 > 21:29:45 CST > &gt;&gt; 2021, Replication Lag=40698192 > &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SINK&nbsp; : > AgeOfLastAppliedOp=927, TimeStampsOfLastAppliedOp=Mon Jan > &gt;&gt; 04 21:30:01 CST 2021 > &gt;&gt; Took 0.0439 seconds > &gt;&gt; > &gt;&gt; > &gt;&gt; =&gt; > #<Java::JavaUtil::Collections::UnmodifiableSet:0x397a10df&gt; > &gt;&gt; > &gt;&gt; > &gt;&gt; > &gt;&gt; ------------------ ???????? ------------------ > &gt;&gt; *??????:* "user-zh" <palomino...@gmail.com&gt;; > &gt;&gt; *????????:* 2021??1??4??(??????) ????9:22 > &gt;&gt; *??????:* "user-zh"<user-zh@hbase.apache.org&gt;; > &gt;&gt; *????:* Re: ????HBase????????HBase2.1.0 ???????????????????????? > &gt;&gt; > &gt;&gt; ????????????????????replication??????????????????????queue??????????replication?????????????? > &gt;&gt; > &gt;&gt; ????????????replication peer?????????? > &gt;&gt; > &gt;&gt; sudo rm -rf /* <2326130...@qq.com&gt; ??2021??1??4?????? ????11:27?????? > &gt;&gt; > &gt;&gt; &gt; hi > &gt;&gt; &gt;&nbsp; > ????????????????????????????????Replication?????????????????????????????????????????????????????????????????????????? > &gt;&gt; &gt; > &gt;&gt; &gt; ????11??21:01.959?? WARN ReplicationSource > &gt;&gt; &gt; > &gt;&gt; &gt; WAL group > &gt;&gt; > centos-bigdata-datanode-10-2-5-200.intsig.internal%2C16020%2C1599665626397.centos-bigdata-datanode-10-2-5-200.intsig.internal%2C16020%2C1599665626397.regiongroup-0 > &gt;&gt; queue size: 86709 exceeds value of > replication.source.log.queue.warn: 2 > &gt;&gt; &gt; > &gt;&gt; &gt; ????11??21:01.979?? WARN FanOutOneBlockAsyncDFSOutputHelper > &gt;&gt; &gt; > &gt;&gt; &gt; complete file > &gt;&gt; > /hbase/WALs/centos-bigdata-datanode-10-2-5-200.intsig.internal,16020,1599665626397/centos-bigdata-datanode-10-2-5-200.intsig.internal%2C16020%2C1599665626397.centos-bigdata-datanode-10-2-5-200.intsig.internal%2C16020%2C1599665626397.regiongroup-0.1609730220231 > &gt;&gt; not finished, retry = 0 > &gt;&gt; &gt; > &gt;&gt; &gt; ????11??21:02.736?? INFO AbstractFSWAL > &gt;&gt; &gt; > &gt;&gt; &gt; Too many WALs; count=16512, max=256; forcing flush of 2 > regions(s): > &gt;&gt; ce7c15393f6021611c12eb8f7739c55a, a9e65042043cf6c122785aee3dd52d98 > &gt;&gt; &gt; > &gt;&gt; &gt; ????11??21:02.736?? WARN LogRoller > &gt;&gt; &gt; > &gt;&gt; &gt; Failed to schedule flush of ce7c15393f6021611c12eb8f7739c55a, > &gt;&gt; region=null, requester=null > &gt;&gt; &gt; > &gt;&gt; &gt; ????11??21:02.736?? WARN LogRoller > &gt;&gt; &gt; > &gt;&gt; &gt; Failed to schedule flush of a9e65042043cf6c122785aee3dd52d98, > &gt;&gt; region=null, requester=null > &gt;&gt; &gt; > &gt;&gt; &gt; > &gt;&gt; &gt; ------------------ ???????? ------------------ > &gt;&gt; &gt; *??????:* "sudo rm -rf /*" <2326130...@qq.com&gt;; > &gt;&gt; &gt; *????????:* 2021??1??4??(??????) ????11:21 > &gt;&gt; &gt; *??????:* "user-zh"<user-zh@hbase.apache.org&gt;; > &gt;&gt; &gt; *????:* "user-zh"<user-zh@hbase.apache.org&gt;; > &gt;&gt; &gt; *????:* ?????? ????HBase????????HBase2.1.0 ???????????????????????? > &gt;&gt; &gt; > &gt;&gt; &gt; hi > &gt;&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; > ??????????????????????????????????????????????????????????????????????????????????????WAL log > &gt;&gt; &gt; > &gt;&gt; > ????????????????????????RegionServer????????????????8??????????????????????oldWALs??????regionserver??????????????????????????????????????????REPLICATION_SCOPE=1?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? > &gt;&gt; &gt; > &gt;&gt; &gt; > &gt;&gt; &gt; ------------------ ???????? ------------------ > &gt;&gt; &gt; *??????:* "user-zh" <palomino...@gmail.com&gt;; > &gt;&gt; &gt; *????????:* 2021??1??4??(??????) ????11:05 > &gt;&gt; &gt; *??????:* "user-zh"<user-zh@hbase.apache.org&gt;; > &gt;&gt; &gt; *????:* Re: ????HBase????????HBase2.1.0 ???????????????????????? > &gt;&gt; &gt; > &gt;&gt; &gt; ???????????????????? > &gt;&gt; &gt; > &gt;&gt; &gt; > ??????????????????????????????????????????????????????oldWALs??????regionserver???????????????????????????????????? > &gt;&gt; &gt; > &gt;&gt; &gt; sudo rm -rf /* <2326130...@qq.com&gt; ??2021??1??4?????? > ????10:45?????? > &gt;&gt; &gt; > &gt;&gt; &gt; &gt; hi, > &gt;&gt; &gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; ????????????????????????HBase replication > ??????????????????????WAL > &gt;&gt; &gt; &gt; log????????????????????????????????ReplicationSource > &gt;&gt; &gt; &gt; ??????????????WATTING??BLOCKED?????????????????????????????????????? > &gt;&gt; &gt; &gt; ???????????????????????? > &gt;&gt; &gt; &gt; > &gt;&gt; &gt; &gt; > &gt;&gt; &gt; &gt; > &gt;&gt; &gt; &gt; ------------------ ???????? ------------------ > &gt;&gt; &gt; &gt; *??????:* "user-zh" <18031...@qq.com&gt;; > &gt;&gt; &gt; &gt; *????????:* 2021??1??2??(??????) ????7:43 > &gt;&gt; &gt; &gt; *??????:* "user-zh"<user-zh@hbase.apache.org&gt;; > &gt;&gt; &gt; &gt; *????:* ?????? ????HBase????????HBase2.1.0 ???????????????????????? > &gt;&gt; &gt; &gt; > &gt;&gt; &gt; &gt; ???????????????????????????????????? > &gt;&gt; &gt; &gt; ??????????????????????????2???????????????????????????????????????? > &gt;&gt; &gt; &gt; > &gt;&gt; &gt; &gt; > &gt;&gt; &gt; &gt; > &gt;&gt; &gt; &gt; > &gt;&gt; &gt; &gt; > ------------------&amp;nbsp;????????&amp;nbsp;------------------ > &gt;&gt; &gt; &gt; ??????: > &gt;&gt; &gt; &gt; "user-zh" > &gt;&gt; &gt; &gt; <2326130...@qq.com&amp;gt;; > &gt;&gt; &gt; &gt; ????????:&amp;nbsp;2020??12??31??(??????) ????2:27 > &gt;&gt; &gt; &gt; ??????:&amp;nbsp;"user-zh"<user-zh@hbase.apache.org > &amp;gt;; > &gt;&gt; &gt; &gt; > &gt;&gt; &gt; &gt; ????:&amp;nbsp;?????? ????HBase????????HBase2.1.0 ???????????????????????? > &gt;&gt; &gt; &gt; > &gt;&gt; &gt; &gt; > &gt;&gt; &gt; &gt; > &gt;&gt; &gt; &gt; hi > &gt;&gt; &gt; &gt; ???????????????????????????????????????????????????????????????????????????????????? > &gt;&gt; &gt; &gt; > &gt;&gt; &gt; &gt; > &gt;&gt; &gt; &gt; > &gt;&gt; &gt; &gt; > &gt;&gt; &gt; &gt; ------------------ ???????? ------------------ > &gt;&gt; &gt; &gt; ??????: > &gt;&gt; &gt; &gt; "user-zh" > &gt;&gt; &gt; &gt; <weic...@cloudera.com.INVALID&amp;gt;; > &gt;&gt; &gt; &gt; ????????:&amp;nbsp;2020??12??31??(??????) ????2:21 > &gt;&gt; &gt; &gt; ??????:&amp;nbsp;"user-zh"<user-zh@hbase.apache.org > &amp;gt;; > &gt;&gt; &gt; &gt; > &gt;&gt; &gt; &gt; ????:&amp;nbsp;Re: ????HBase????????HBase2.1.0 ???????????????????????? > &gt;&gt; &gt; &gt; > &gt;&gt; &gt; &gt; > &gt;&gt; &gt; &gt; > &gt;&gt; &gt; &gt; ?D???????????o???@?? ???????l???? > &gt;&gt; &gt; &gt; > &gt;&gt; &gt; &gt; On Wed, Dec 30, 2020 at 7:05 PM sudo rm -rf /* < > 2326130...@qq.com&amp;gt; > &gt;&gt; &gt; &gt; wrote: > &gt;&gt; &gt; &gt; > &gt;&gt; &gt; &gt; &amp;gt; hi > &gt;&gt; &gt; &gt; &amp;gt; > &gt;&gt; &gt; &gt; &amp;gt; ?????????????????????????????????????????? > &gt;&gt; &gt; &gt; &amp;gt; > &gt;&gt; &gt; &gt; &amp;gt; > ????????HBase????????????????????????????replication??????????????????????????????????????HBase???? > &gt;&gt; &gt; &gt; &amp;gt; hbase2.1.0-cdh-6.3.2 > &gt;&gt; &gt; &gt; &amp;gt; > &gt;&gt; &gt; &gt; &amp;gt; ?????????????????????????? > &gt;&gt; &gt; &gt; &amp;gt; > &gt;&gt; &gt; &gt; &amp;gt; ??????replication > ??????replication??????????????????????????????????????WAL > &gt;&gt; &gt; &gt; Log????replication?????????????? > &gt;&gt; &gt; &gt; &amp;gt; > &gt;&gt; &gt; &gt; &amp;gt; > &gt;&gt; &gt; &gt; &amp;gt; > &gt;&gt; &gt; &gt; &amp;gt; > &gt;&gt; &gt; &gt; &amp;gt; > &gt;&gt; &gt; &gt; &amp;gt; ??????????replication???????????????????? > &gt;&gt; &gt; &gt; &amp;gt; > &gt;&gt; &gt; &gt; &amp;gt; ????????????WALs oldWals???????????? > &gt;&gt; &gt; &gt; &amp;gt; > &gt;&gt; &gt; &gt; &amp;gt; ?????? > &gt;&gt; &gt; &gt; &amp;gt; > &gt;&gt; &gt; &gt; &amp;gt; > &gt;&gt; &gt; &gt; &amp;gt; ?????? > &gt;&gt; &gt; &gt; &amp;gt; > &gt;&gt; &gt; &gt; &amp;gt; > &gt;&gt; &gt; &gt; &amp;gt; Replication???????????????????? > &gt;&gt; &gt; &gt; &amp;gt; > &gt;&gt; &gt; &gt; &amp;gt; > ??????????????????????????????????????????????????????????????????replication??BUG?????????????????????? > &gt;&gt; &gt; &gt; &amp;gt; > &gt;&gt; &gt; &gt; &amp;gt; > &gt;&gt; &gt; &gt; &amp;gt; > &gt;&gt; &gt; &gt; > &gt;&gt; &gt; > &gt;&gt; > &gt;