Thanks for answering the queries.
---
Mallikarjun
On Wed, Aug 18, 2021 at 9:32 AM 张铎(Duo Zhang) <palomino...@gmail.com> wrote:
> In hbase, the mvcc write number and wal sequence id are the same
> thing, so when we just want to bump the mvcc number, we will not write
> an actual WAL but the sequence id will be increased.
>
> And I think it will be good to have a MR job to replicate WAL serially.
>
> Mallikarjun <mallik.v.ar...@gmail.com> 于2021年8月18日周三 上午10:50写道：
> >
> > Inline Reply
> >
> > On Wed, Aug 18, 2021 at 8:06 AM 张铎(Duo Zhang) <palomino...@gmail.com>
> wrote:
> >
> > > Mallikarjun <mallik.v.ar...@gmail.com> 于2021年8月18日周三 上午10:19写道：
> > > >
> > > > Thanks for the response @Duo
> > > >
> > > > Inline reply
> > > >
> > > > On Wed, Aug 18, 2021 at 7:37 AM 张铎(Duo Zhang) <palomino...@gmail.com
> >
> > > wrote:
> > > >
> > > > > This is the isRangeFinished method
> > > > >
> > > > > private boolean isRangeFinished(long endBarrier, String
> > > > > encodedRegionName) throws IOException {
> > > > > long pushedSeqId;
> > > > > try {
> > > > > pushedSeqId = storage.getLastSequenceId(encodedRegionName,
> > > peerId);
> > > > > } catch (ReplicationException e) {
> > > > > throw new IOException(
> > > > > "Failed to get pushed sequence id for " +
> encodedRegionName +
> > > > > ", peer " + peerId, e);
> > > > > }
> > > > > // endBarrier is the open sequence number. When opening a
> region,
> > > > > the open sequence number will
> > > > > // be set to the old max sequence id plus one, so here we need
> to
> > > > > minus one.
> > > > > return pushedSeqId >= endBarrier - 1;
> > > > > }
> > > > >
> > > > > So for this region
> > > > >
> > > > > rs-9 24c765b42253f96b550831d83e99cc9e 18775105 18762209 [17776286,
> +++
> > > > > 18762210, 18775053, 18775079, 18775104, -- 18775119]
> > > > >
> > > > > We have already finished the first range [17776286, 18762210), but
> > > > > then we jump directly to range [18775053, 18775079), so the problem
> > > > > here is where is the [18762210, 18775053)...
> > > > >
> > > >
> > > > These sequence ID's are present in WAL's which are not cleaned up.
> (in
> > > > OLDWALs)
> > > >
> > > > Related Question: Is it allowed to have gaps in sequence IDs in WAL's
> > > for a
> > > > single region?
> > > > Example: for Region: *24c765b42253f96b550831d83e99cc9e*, if
> sequence ID
> > > > *18775105* is present, can I expect *18775106 *is mandatory to be
> > > present?
> > > > or there can be gaps.
> > > Inside a range it is allowed to have gaps, but when reopening a
> > > region, we need to make sure there are no gaps otherwise the
> > > replication will be stuck.
> > >
> >
> > Just curious to know, what are some scenarios which can lead to gaps?
> From
> > my small number of experiments It was consecutive in nature, I did not
> find
> > such gap scenarios.
> >
> >
> > > >
> > > >
> > > > >
> > > > > And on the fix, you can clear the range information for the given
> > > > > regions in meta table, and then restart the clusters, I think the
> > > > > replication could continue.
> > > > >
> > > > >
> > > > If you mean removing some barriers so that replication is unblocked.
> > > > Doesn't it lead to *out of order events *replicated end up in
> > > > corrupting data?
> > > Yes, the replication will be out of order. But this is the easier way
> > > to recover the replication.
> > > If you still want to obtain the order, then you need to find out the
> > > root cause of my question, where is the WAL for the missing ranges. Is
> > > it because we have already replicated the data but do not mark the
> > > range as finished, or we just lose the WAL data for the range?
> > >
> >
> > Current scenario occurred in Active Passive cluster setup and replication
> > is stuck on the Passive side. So I won't be able to answer following
> > question
> >
> > we have already replicated the data
> >
> >
> > But the following comment can help me find data from oldWAL if last
> > sequence id is present or not before region movement.
> >
> > but when reopening a region, we need to make sure there are no gaps
> > >
> >
> > Alternatively, do you think it is a good idea to write a job similar
> > to *RecoveredReplicationSource
> > *which ensures serial replication to other cluster outside of hbase
> cluster?
> >
> >
> > > >
> > > >
> > > > > Mallikarjun <mallik.v.ar...@gmail.com> 于2021年8月17日周二 下午3:04写道：
> > > > > >
> > > > > > I have got into the following scenario. I won't go into details
> of
> > > how I
> > > > > > got here, since I am not able to reliably reproduce this scenario
> > > thus
> > > > > far.
> > > > > > (Typically happens when some rs goes down because of hardware
> issues)
> > > > > >
> > > > > > Let me explain to you the following details.
> > > > > > Col 1: Region server on which region is trying to replicate
> > > > > > Col 2: Region trying to replicate but stuck
> > > > > > Col 3: SequenceID which is being replicated and stuck because
> > > previous
> > > > > > range is not finished
> > > > > > Col 4: Checkpoint in zk until which sequence id is already
> > > replicated to
> > > > > > peer
> > > > > > Col 5: Replication barriers for that region. This is a list of
> open
> > > > > > sequence IDs on region movement. (+++ means where *checkpoint*
> > > belongs,
> > > > > ---
> > > > > > is where *to replicate seqid* belongs)
> > > > > >
> > > > > > There are in total 53 regions and 10 regionservers
> > > > > >
> > > > > > RegionServer Region Trying to replicate sequenceID Replicated
> until
> > > > > Current
> > > > > > Barriers
> > > > > > rs-9 24c765b42253f96b550831d83e99cc9e 18775105 18762209
> [17776286,
> > > +++
> > > > > > 18762210, 18775053, 18775079, 18775104, -- 18775119]
> > > > > > rs-5 b4144bfe75c5826710ec54849741b038 189154192 189091221
> > > [184183678, +++
> > > > > > 189117430, 189154191, -- 189154327]
> > > > > > rs-8 deb6fee3380e7b9db9826cb5f27f8a59 189099509 189036510
> > > [180662218, +++
> > > > > > 189062798, 189099508, -- 189099587]
> > > > > > rs-8 3338fd34ae7ba06a7eccd89048fa83ce 189078951 189077722
> > > [184170310, +++
> > > > > > 189078876, 189078950, -- 189104780, 189141509, 189141545,
> 189141595]
> > > > > > rs-6 1af22c68b9212971ab2570e14b7b0dc2 183301002 183265047
> > > [180239864, +++
> > > > > > 183265048, 183270357, 183277363, 183300886, 183301001, --
> 183301062]
> > > > > > rs-10 1af22c68b9212971ab2570e14b7b0dc2 183301063 183265047
> > > [180239864,
> > > > > +++
> > > > > > 183265048, 183270357, 183277363, 183300886, 183301001, 183301062
> --]
> > > > > > rs-6 4b9e98c7eca7a24c74136de1aa8aeab0 189027036 189022619
> > > [189022618, +++
> > > > > > 189027035, 189085155, 189085241, 189085290]
> > > > > > rs-4 e45ba292df95edbdf884e2ec50cf5f16 189099081 189062191
> > > [184126535, +++
> > > > > > 189098947, 189099080, -- 189099226]
> > > > > > rs-4 83e65729dcad644738a0a3cee994e2df 189012454 189012365
> > > [184103269, +++
> > > > > > 189012453, -- 189012538, 189074967, 189075016, 189075294,
> 189075349]
> > > > > > rs-10 83e65729dcad644738a0a3cee994e2df 189012539 189012365
> > > [184103269,
> > > > > +++
> > > > > > 189012453, 189012538, -- 189074967, 189075016, 189075294,
> 189075349]
> > > > > > rs-3 11fca95de4878782af53371a25cf44d0 189121426 189058129
> > > [180684344, +++
> > > > > > 189084916, 189121283, 189121425, -- 189121602]
> > > > > > rs-3 b9db001578e127740d7e0e186e4fbab6 189145458 189081436
> > > [184175242, +++
> > > > > > 189083026, 189145417, 189145457, -- 189145562, 189145723,
> 189145781]
> > > > > > rs-2 262ca9ff7b878f32c451fac3eb430a88 189128535 189065879
> > > [184159187, +++
> > > > > > 189091684, 189128534, -- 189128708]
> > > > > > rs-2 03a1eb906a344944aad727dbb8210cfc 172392082 172390331
> > > [167737983, +++
> > > > > > 172392081, -- 172400093, 172446121, 172446172]
> > > > > > rs-10 ae2726c7b4eeec3f93336d71e80145a4 189027430 189026939
> > > [184119428,
> > > > > +++
> > > > > > 189027429, -- 189053118, 189089933, 189089995, 189090059]
> > > > > > rs-10 770ba4f4568fff803e6df340b2ffe486 189034144 189032879
> > > [184127026,
> > > > > +++
> > > > > > 189034143, --189048295, 189059834, 189096413, 189096513,
> 189096548,
> > > > > > 189096606]
> > > > > > rs-1 5846f4ce8acdd5aabf325c847d18c729 18793501 18780639
> [18778549,
> > > +++
> > > > > > 18784783, 18793471, 18793484, 18793500 --]
> > > > > > rs-1 5846f4ce8acdd5aabf325c847d18c729 18793472 18780639
> [18778549,
> > > +++
> > > > > > 18784783, 18793471, --- 18793484, 18793500]
> > > > > > rs-1 fabd3ea591d5f20a86a26f8767d34f63 189028498 189024357
> > > [184116531, +++
> > > > > > 189025318, 189028497, --- 189051176, 189087488, 189087737,
> 189087850]
> > > > > > rs-1 335d855c5005343719ea73bcb7dcb269 189064849 189037338
> > > [184130122, +++
> > > > > > 189064848, --- 189101485, 189101698, 189101774]
> > > > > >
> > > > > >
> > > > > > My question is, how do I recover from here? Any suggestions.
> > > > > >
> > > > > > Only thought is that I have to replay by writing some MR jobs /
> some
> > > > > > scripts to read and replay selectively and update checkpoints.
> > > > > >
> > > > > > ---
> > > > > > Mallikarjun
> > > > >
> > >
>
