[jira] [Commented] (HBASE-29463) Bidirectional serial replication will block if a region’s last edit before rs crashed was from the peer cluster
[
https://issues.apache.org/jira/browse/HBASE-29463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18014343#comment-18014343
]
Hudson commented on HBASE-29463:
Results for branch branch-3
[build #453 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/453/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/453/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk17 hadoop3 checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/453/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk17 hadoop 3.3.5 backward compatibility checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/453/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk17 hadoop 3.3.6 backward compatibility checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/453/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk17 hadoop 3.4.0 backward compatibility checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/453/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test for 3.3.5 {color}
(/) {color:green}+1 client integration test for 3.3.6 {color}
(/) {color:green}+1 client integration test for 3.4.0 {color}
(/) {color:green}+1 client integration test for 3.4.1 {color}
> Bidirectional serial replication will block if a region’s last edit before rs
> crashed was from the peer cluster
> ---
>
> Key: HBASE-29463
> URL: https://issues.apache.org/jira/browse/HBASE-29463
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.4.5
>Reporter: haosen chen
>Assignee: Duo Zhang
>Priority: Critical
> Labels: pull-request-available
> Fix For: 2.7.0, 3.0.0-beta-2, 2.6.4, 2.5.13
>
> Attachments: image-2025-07-21-14-52-19-057.png,
> image-2025-07-21-14-52-47-751.png
>
>
> For two HBase clusters that enable bidirectional replication and set up
> serial replication, when a region in cluster A received last edit from peer
> cluster before RS crashed, the replication from cluster A to B will block.
> Because in this situation, the HBase replication system will wait until the
> last pushed sequence id reaches the new barrier but edit received from peer
> cluster will never be pushed.
> When Region r1 in Cluster A pushes its last edit (e.g., seqID 58) to Cluster
> B and subsequently received two additional edits (seqID 59–60) from Cluster B
> and then the rs crashed, Region r1 will be reopened on another RegionServer
> and set a new barrier at seqID 61. However, edits 59–60 will never be pushed
> to Cluster B again, causing the _last pushed sequenceId_ to stagnate. As a
> result, the {{SerialReplicationChecker}} will repeatedly fail its checks.
> The new RS will keep print DEBUG LOG:
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Replication barrier for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>]:
> ReplicationBarrierResult [barriers=[23, 29, 68], state=OPEN,
> parentRegionNames=]
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Previous range for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>] has not been
> finished yet, give up
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Can not push
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>], wait
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-29463) Bidirectional serial replication will block if a region’s last edit before rs crashed was from the peer cluster
[
https://issues.apache.org/jira/browse/HBASE-29463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18014335#comment-18014335
]
Hudson commented on HBASE-29463:
Results for branch master
[build #1333 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1333/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1333/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk17 hadoop3 checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1333/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk17 hadoop 3.3.5 backward compatibility checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1333/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk17 hadoop 3.3.6 backward compatibility checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1333/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk17 hadoop 3.4.0 backward compatibility checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1333/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test for 3.3.5 {color}
(/) {color:green}+1 client integration test for 3.3.6 {color}
(/) {color:green}+1 client integration test for 3.4.0 {color}
(/) {color:green}+1 client integration test for 3.4.1 {color}
> Bidirectional serial replication will block if a region’s last edit before rs
> crashed was from the peer cluster
> ---
>
> Key: HBASE-29463
> URL: https://issues.apache.org/jira/browse/HBASE-29463
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.4.5
>Reporter: haosen chen
>Assignee: Duo Zhang
>Priority: Critical
> Labels: pull-request-available
> Fix For: 2.7.0, 3.0.0-beta-2, 2.6.4, 2.5.13
>
> Attachments: image-2025-07-21-14-52-19-057.png,
> image-2025-07-21-14-52-47-751.png
>
>
> For two HBase clusters that enable bidirectional replication and set up
> serial replication, when a region in cluster A received last edit from peer
> cluster before RS crashed, the replication from cluster A to B will block.
> Because in this situation, the HBase replication system will wait until the
> last pushed sequence id reaches the new barrier but edit received from peer
> cluster will never be pushed.
> When Region r1 in Cluster A pushes its last edit (e.g., seqID 58) to Cluster
> B and subsequently received two additional edits (seqID 59–60) from Cluster B
> and then the rs crashed, Region r1 will be reopened on another RegionServer
> and set a new barrier at seqID 61. However, edits 59–60 will never be pushed
> to Cluster B again, causing the _last pushed sequenceId_ to stagnate. As a
> result, the {{SerialReplicationChecker}} will repeatedly fail its checks.
> The new RS will keep print DEBUG LOG:
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Replication barrier for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>]:
> ReplicationBarrierResult [barriers=[23, 29, 68], state=OPEN,
> parentRegionNames=]
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Previous range for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>] has not been
> finished yet, give up
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Can not push
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>], wait
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-29463) Bidirectional serial replication will block if a region’s last edit before rs crashed was from the peer cluster
[
https://issues.apache.org/jira/browse/HBASE-29463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18014249#comment-18014249
]
Hudson commented on HBASE-29463:
Results for branch branch-2.6
[build #349 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/349/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/349/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/349/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/349/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/349/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk17 hadoop3 checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/349/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk17 hadoop 3.3.5 backward compatibility checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/349/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk17 hadoop 3.3.6 backward compatibility checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/349/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk17 hadoop 3.4.0 backward compatibility checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/349/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test for HBase 2 {color}
(/) {color:green}+1 client integration test for 3.3.5 {color}
(/) {color:green}+1 client integration test for 3.3.6 {color}
(/) {color:green}+1 client integration test for 3.4.0 {color}
(/) {color:green}+1 client integration test for 3.4.1 {color}
> Bidirectional serial replication will block if a region’s last edit before rs
> crashed was from the peer cluster
> ---
>
> Key: HBASE-29463
> URL: https://issues.apache.org/jira/browse/HBASE-29463
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.4.5
>Reporter: haosen chen
>Assignee: Duo Zhang
>Priority: Critical
> Labels: pull-request-available
> Fix For: 2.7.0, 3.0.0-beta-2, 2.6.4, 2.5.13
>
> Attachments: image-2025-07-21-14-52-19-057.png,
> image-2025-07-21-14-52-47-751.png
>
>
> For two HBase clusters that enable bidirectional replication and set up
> serial replication, when a region in cluster A received last edit from peer
> cluster before RS crashed, the replication from cluster A to B will block.
> Because in this situation, the HBase replication system will wait until the
> last pushed sequence id reaches the new barrier but edit received from peer
> cluster will never be pushed.
> When Region r1 in Cluster A pushes its last edit (e.g., seqID 58) to Cluster
> B and subsequently received two additional edits (seqID 59–60) from Cluster B
> and then the rs crashed, Region r1 will be reopened on another RegionServer
> and set a new barrier at seqID 61. However, edits 59–60 will never be pushed
> to Cluster B again, causing the _last pushed sequenceId_ to stagnate. As a
> result, the {{SerialReplicationChecker}} will repeatedly fail its checks.
> The new RS will keep print DEBUG LOG:
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Replication barrier for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>]:
> ReplicationBarrierResult [barriers=[23, 29, 68], state=OPEN,
> parentRegionNames=]
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Previous range for
>
[jira] [Commented] (HBASE-29463) Bidirectional serial replication will block if a region’s last edit before rs crashed was from the peer cluster
[
https://issues.apache.org/jira/browse/HBASE-29463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18014216#comment-18014216
]
Hudson commented on HBASE-29463:
Results for branch branch-2.5
[build #723 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/723/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/723/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/723/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/723/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/723/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk17 hadoop3 checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/723/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk17 hadoop 3.2.4 backward compatibility checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/723/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk17 hadoop 3.3.5 backward compatibility checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/723/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk17 hadoop 3.3.6 backward compatibility checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/723/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 source release artifact{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/723//console].
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/723//console].
> Bidirectional serial replication will block if a region’s last edit before rs
> crashed was from the peer cluster
> ---
>
> Key: HBASE-29463
> URL: https://issues.apache.org/jira/browse/HBASE-29463
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.4.5
>Reporter: haosen chen
>Assignee: Duo Zhang
>Priority: Critical
> Labels: pull-request-available
> Fix For: 2.7.0, 3.0.0-beta-2, 2.6.4, 2.5.13
>
> Attachments: image-2025-07-21-14-52-19-057.png,
> image-2025-07-21-14-52-47-751.png
>
>
> For two HBase clusters that enable bidirectional replication and set up
> serial replication, when a region in cluster A received last edit from peer
> cluster before RS crashed, the replication from cluster A to B will block.
> Because in this situation, the HBase replication system will wait until the
> last pushed sequence id reaches the new barrier but edit received from peer
> cluster will never be pushed.
> When Region r1 in Cluster A pushes its last edit (e.g., seqID 58) to Cluster
> B and subsequently received two additional edits (seqID 59–60) from Cluster B
> and then the rs crashed, Region r1 will be reopened on another RegionServer
> and set a new barrier at seqID 61. However, edits 59–60 will never be pushed
> to Cluster B again, causing the _last pushed sequenceId_ to stagnate. As a
> result, the {{SerialReplicationChecker}} will repeatedly fail its checks.
> The new RS will keep print DEBUG LOG:
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Replication barrier for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>]:
> ReplicationBarrierResult [barriers=[23, 29, 68], state=OPEN,
> parentRegionNames=]
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Previous range for
>
[jira] [Commented] (HBASE-29463) Bidirectional serial replication will block if a region’s last edit before rs crashed was from the peer cluster
[
https://issues.apache.org/jira/browse/HBASE-29463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18014080#comment-18014080
]
Hudson commented on HBASE-29463:
Results for branch branch-2
[build #1309 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1309/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1309/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1309/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1309/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1309/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk17 hadoop3 checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1309/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk17 hadoop 3.3.5 backward compatibility checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1309/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk17 hadoop 3.3.6 backward compatibility checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1309/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk17 hadoop 3.4.0 backward compatibility checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1309/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test for HBase 2 {color}
(/) {color:green}+1 client integration test for 3.3.5 {color}
(/) {color:green}+1 client integration test for 3.3.6 {color}
(/) {color:green}+1 client integration test for 3.4.0 {color}
(/) {color:green}+1 client integration test for 3.4.1 {color}
> Bidirectional serial replication will block if a region’s last edit before rs
> crashed was from the peer cluster
> ---
>
> Key: HBASE-29463
> URL: https://issues.apache.org/jira/browse/HBASE-29463
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.4.5
>Reporter: haosen chen
>Assignee: Duo Zhang
>Priority: Critical
> Labels: pull-request-available
> Fix For: 2.7.0, 3.0.0-beta-2, 2.6.4, 2.5.13
>
> Attachments: image-2025-07-21-14-52-19-057.png,
> image-2025-07-21-14-52-47-751.png
>
>
> For two HBase clusters that enable bidirectional replication and set up
> serial replication, when a region in cluster A received last edit from peer
> cluster before RS crashed, the replication from cluster A to B will block.
> Because in this situation, the HBase replication system will wait until the
> last pushed sequence id reaches the new barrier but edit received from peer
> cluster will never be pushed.
> When Region r1 in Cluster A pushes its last edit (e.g., seqID 58) to Cluster
> B and subsequently received two additional edits (seqID 59–60) from Cluster B
> and then the rs crashed, Region r1 will be reopened on another RegionServer
> and set a new barrier at seqID 61. However, edits 59–60 will never be pushed
> to Cluster B again, causing the _last pushed sequenceId_ to stagnate. As a
> result, the {{SerialReplicationChecker}} will repeatedly fail its checks.
> The new RS will keep print DEBUG LOG:
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Replication barrier for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>]:
> ReplicationBarrierResult [barriers=[23, 29, 68], state=OPEN,
> parentRegionNames=]
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Previous range for
>
[jira] [Commented] (HBASE-29463) Bidirectional serial replication will block if a region’s last edit before rs crashed was from the peer cluster
[
https://issues.apache.org/jira/browse/HBASE-29463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18013206#comment-18013206
]
haosen chen commented on HBASE-29463:
-
[~zhangduo] The PR seems good. I have verified it using two HBase 2.4.5
clusters and it does fix this bug. Thanks!!
> Bidirectional serial replication will block if a region’s last edit before rs
> crashed was from the peer cluster
> ---
>
> Key: HBASE-29463
> URL: https://issues.apache.org/jira/browse/HBASE-29463
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.4.5
>Reporter: haosen chen
>Assignee: Duo Zhang
>Priority: Critical
> Labels: pull-request-available
> Attachments: image-2025-07-21-14-52-19-057.png,
> image-2025-07-21-14-52-47-751.png
>
>
> For two HBase clusters that enable bidirectional replication and set up
> serial replication, when a region in cluster A received last edit from peer
> cluster before RS crashed, the replication from cluster A to B will block.
> Because in this situation, the HBase replication system will wait until the
> last pushed sequence id reaches the new barrier but edit received from peer
> cluster will never be pushed.
> When Region r1 in Cluster A pushes its last edit (e.g., seqID 58) to Cluster
> B and subsequently received two additional edits (seqID 59–60) from Cluster B
> and then the rs crashed, Region r1 will be reopened on another RegionServer
> and set a new barrier at seqID 61. However, edits 59–60 will never be pushed
> to Cluster B again, causing the _last pushed sequenceId_ to stagnate. As a
> result, the {{SerialReplicationChecker}} will repeatedly fail its checks.
> The new RS will keep print DEBUG LOG:
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Replication barrier for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>]:
> ReplicationBarrierResult [barriers=[23, 29, 68], state=OPEN,
> parentRegionNames=]
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Previous range for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>] has not been
> finished yet, give up
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Can not push
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>], wait
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-29463) Bidirectional serial replication will block if a region’s last edit before rs crashed was from the peer cluster
[
https://issues.apache.org/jira/browse/HBASE-29463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18010136#comment-18010136
]
Duo Zhang commented on HBASE-29463:
---
[~haosen chen] The PR is ready, please take a look and also try if it can solve
your problem.
Thanks.
> Bidirectional serial replication will block if a region’s last edit before rs
> crashed was from the peer cluster
> ---
>
> Key: HBASE-29463
> URL: https://issues.apache.org/jira/browse/HBASE-29463
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.4.5
>Reporter: haosen chen
>Assignee: Duo Zhang
>Priority: Critical
> Labels: pull-request-available
> Attachments: image-2025-07-21-14-52-19-057.png,
> image-2025-07-21-14-52-47-751.png
>
>
> For two HBase clusters that enable bidirectional replication and set up
> serial replication, when a region in cluster A received last edit from peer
> cluster before RS crashed, the replication from cluster A to B will block.
> Because in this situation, the HBase replication system will wait until the
> last pushed sequence id reaches the new barrier but edit received from peer
> cluster will never be pushed.
> When Region r1 in Cluster A pushes its last edit (e.g., seqID 58) to Cluster
> B and subsequently received two additional edits (seqID 59–60) from Cluster B
> and then the rs crashed, Region r1 will be reopened on another RegionServer
> and set a new barrier at seqID 61. However, edits 59–60 will never be pushed
> to Cluster B again, causing the _last pushed sequenceId_ to stagnate. As a
> result, the {{SerialReplicationChecker}} will repeatedly fail its checks.
> The new RS will keep print DEBUG LOG:
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Replication barrier for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>]:
> ReplicationBarrierResult [barriers=[23, 29, 68], state=OPEN,
> parentRegionNames=]
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Previous range for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>] has not been
> finished yet, give up
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Can not push
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>], wait
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-29463) Bidirectional serial replication will block if a region’s last edit before rs crashed was from the peer cluster
[
https://issues.apache.org/jira/browse/HBASE-29463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18009609#comment-18009609
]
Duo Zhang commented on HBASE-29463:
---
Before actually fixing the problem, let me provide a UT to reproduce the
problem first.
And on the fixing way, I think we need to have seprated logic to determine
whether we need to record the last pushed sequence id in the batch, even if the
WALEntry is filtered out.
> Bidirectional serial replication will block if a region’s last edit before rs
> crashed was from the peer cluster
> ---
>
> Key: HBASE-29463
> URL: https://issues.apache.org/jira/browse/HBASE-29463
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.4.5
>Reporter: haosen chen
>Assignee: Duo Zhang
>Priority: Critical
> Attachments: image-2025-07-21-14-52-19-057.png,
> image-2025-07-21-14-52-47-751.png
>
>
> For two HBase clusters that enable bidirectional replication and set up
> serial replication, when a region in cluster A received last edit from peer
> cluster before RS crashed, the replication from cluster A to B will block.
> Because in this situation, the HBase replication system will wait until the
> last pushed sequence id reaches the new barrier but edit received from peer
> cluster will never be pushed.
> When Region r1 in Cluster A pushes its last edit (e.g., seqID 58) to Cluster
> B and subsequently received two additional edits (seqID 59–60) from Cluster B
> and then the rs crashed, Region r1 will be reopened on another RegionServer
> and set a new barrier at seqID 61. However, edits 59–60 will never be pushed
> to Cluster B again, causing the _last pushed sequenceId_ to stagnate. As a
> result, the {{SerialReplicationChecker}} will repeatedly fail its checks.
> The new RS will keep print DEBUG LOG:
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Replication barrier for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>]:
> ReplicationBarrierResult [barriers=[23, 29, 68], state=OPEN,
> parentRegionNames=]
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Previous range for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>] has not been
> finished yet, give up
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Can not push
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>], wait
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-29463) Bidirectional serial replication will block if a region’s last edit before rs crashed was from the peer cluster
[
https://issues.apache.org/jira/browse/HBASE-29463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18008612#comment-18008612
]
haosen chen commented on HBASE-29463:
-
Yes! If the region goes offline normally, RS will record a close event and
trigger a push
> Bidirectional serial replication will block if a region’s last edit before rs
> crashed was from the peer cluster
> ---
>
> Key: HBASE-29463
> URL: https://issues.apache.org/jira/browse/HBASE-29463
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.4.5
>Reporter: haosen chen
>Priority: Critical
> Attachments: image-2025-07-21-14-52-19-057.png,
> image-2025-07-21-14-52-47-751.png
>
>
> For two HBase clusters that enable bidirectional replication and set up
> serial replication, when a region in cluster A received last edit from peer
> cluster before RS crashed, the replication from cluster A to B will block.
> Because in this situation, the HBase replication system will wait until the
> last pushed sequence id reaches the new barrier but edit received from peer
> cluster will never be pushed.
> When Region r1 in Cluster A pushes its last edit (e.g., seqID 58) to Cluster
> B and subsequently received two additional edits (seqID 59–60) from Cluster B
> and then the rs crashed, Region r1 will be reopened on another RegionServer
> and set a new barrier at seqID 61. However, edits 59–60 will never be pushed
> to Cluster B again, causing the _last pushed sequenceId_ to stagnate. As a
> result, the {{SerialReplicationChecker}} will repeatedly fail its checks.
> The new RS will keep print DEBUG LOG:
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Replication barrier for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>]:
> ReplicationBarrierResult [barriers=[23, 29, 68], state=OPEN,
> parentRegionNames=]
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Previous range for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>] has not been
> finished yet, give up
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Can not push
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>], wait
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-29463) Bidirectional serial replication will block if a region’s last edit before rs crashed was from the peer cluster
[
https://issues.apache.org/jira/browse/HBASE-29463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18008610#comment-18008610
]
Duo Zhang commented on HBASE-29463:
---
{code}
if (doFiltering) {
entry = filterEntry(entry);
}
if (entry != null) {
int sleepMultiplier = 1;
try {
if (!checker.canPush(entry, firstCellInEntryBeforeFiltering)) {
if (batch.getLastWalPosition() > positionBefore) {
// we have something that can push, break
break;
} else {
checker.waitUntilCanPush(entry, firstCellInEntryBeforeFiltering);
}
}
} catch (IOException e) {
LOG.warn("failed to check whether we can push the WAL entries", e);
if (batch.getLastWalPosition() > positionBefore) {
// we have something that can push, break
break;
}
sleepMultiplier = sleep(sleepMultiplier);
}
// arrive here means we can push the entry, record the last sequence id
batch.setLastSeqId(Bytes.toString(entry.getKey().getEncodedRegionName()),
entry.getKey().getSequenceId());
// actually remove the entry.
removeEntryFromStream(entryStream, batch);
if (addEntryToBatch(batch, entry)) {
break;
}
} else {
// actually remove the entry.
removeEntryFromStream(entryStream, batch);
}
{code}
So the problem here is that, we can only record last pushed sequence id when we
actually pushed something out, so if the last entry in a WAL file does not need
to be replicated, we will be stuck.
In the code above, the problem is that, the entry is filtered out at the
beginning, and finally we will enter the else block at the end, without
modifying the last pushed sequence id.
Thanks for finding out this. This is a real problem. Let's think how to fix it.
> Bidirectional serial replication will block if a region’s last edit before rs
> crashed was from the peer cluster
> ---
>
> Key: HBASE-29463
> URL: https://issues.apache.org/jira/browse/HBASE-29463
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.4.5
>Reporter: haosen chen
>Priority: Major
> Attachments: image-2025-07-21-14-52-19-057.png,
> image-2025-07-21-14-52-47-751.png
>
>
> For two HBase clusters that enable bidirectional replication and set up
> serial replication, when a region in cluster A received last edit from peer
> cluster before RS crashed, the replication from cluster A to B will block.
> Because in this situation, the HBase replication system will wait until the
> last pushed sequence id reaches the new barrier but edit received from peer
> cluster will never be pushed.
> When Region r1 in Cluster A pushes its last edit (e.g., seqID 58) to Cluster
> B and subsequently received two additional edits (seqID 59–60) from Cluster B
> and then the rs crashed, Region r1 will be reopened on another RegionServer
> and set a new barrier at seqID 61. However, edits 59–60 will never be pushed
> to Cluster B again, causing the _last pushed sequenceId_ to stagnate. As a
> result, the {{SerialReplicationChecker}} will repeatedly fail its checks.
> The new RS will keep print DEBUG LOG:
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Replication barrier for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>]:
> ReplicationBarrierResult [barriers=[23, 29, 68], state=OPEN,
> parentRegionNames=]
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Previous range for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>] has not been
> finished yet, give up
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Can not push
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>], wait
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-29463) Bidirectional serial replication will block if a region’s last edit before rs crashed was from the peer cluster
[
https://issues.apache.org/jira/browse/HBASE-29463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18008608#comment-18008608
]
Duo Zhang commented on HBASE-29463:
---
But we should increase the last pushed sequence id in this case? This is the
problem?
> Bidirectional serial replication will block if a region’s last edit before rs
> crashed was from the peer cluster
> ---
>
> Key: HBASE-29463
> URL: https://issues.apache.org/jira/browse/HBASE-29463
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.4.5
>Reporter: haosen chen
>Priority: Major
> Attachments: image-2025-07-21-14-52-19-057.png,
> image-2025-07-21-14-52-47-751.png
>
>
> For two HBase clusters that enable bidirectional replication and set up
> serial replication, when a region in cluster A received last edit from peer
> cluster before RS crashed, the replication from cluster A to B will block.
> Because in this situation, the HBase replication system will wait until the
> last pushed sequence id reaches the new barrier but edit received from peer
> cluster will never be pushed.
> When Region r1 in Cluster A pushes its last edit (e.g., seqID 58) to Cluster
> B and subsequently received two additional edits (seqID 59–60) from Cluster B
> and then the rs crashed, Region r1 will be reopened on another RegionServer
> and set a new barrier at seqID 61. However, edits 59–60 will never be pushed
> to Cluster B again, causing the _last pushed sequenceId_ to stagnate. As a
> result, the {{SerialReplicationChecker}} will repeatedly fail its checks.
> The new RS will keep print DEBUG LOG:
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Replication barrier for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>]:
> ReplicationBarrierResult [barriers=[23, 29, 68], state=OPEN,
> parentRegionNames=]
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Previous range for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>] has not been
> finished yet, give up
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Can not push
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>], wait
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-29463) Bidirectional serial replication will block if a region’s last edit before rs crashed was from the peer cluster
[
https://issues.apache.org/jira/browse/HBASE-29463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18008605#comment-18008605
]
haosen chen commented on HBASE-29463:
-
[~zhangduo] Sure. As edit 59-60 are generated within Cluster B, their
clusterIds inherently contain the ID of Cluster B, thereby causing them to be
filtered by the ClusterMarkingEntryFilter.
> Bidirectional serial replication will block if a region’s last edit before rs
> crashed was from the peer cluster
> ---
>
> Key: HBASE-29463
> URL: https://issues.apache.org/jira/browse/HBASE-29463
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.4.5
>Reporter: haosen chen
>Priority: Major
>
> For two HBase clusters that enable bidirectional replication and set up
> serial replication, when a region in cluster A received last edit from peer
> cluster before RS crashed, the replication from cluster A to B will block.
> Because in this situation, the HBase replication system will wait until the
> last pushed sequence id reaches the new barrier but edit received from peer
> cluster will never be pushed.
> When Region r1 in Cluster A pushes its last edit (e.g., seqID 58) to Cluster
> B and subsequently received two additional edits (seqID 59–60) from Cluster B
> and then the rs crashed, Region r1 will be reopened on another RegionServer
> and set a new barrier at seqID 61. However, edits 59–60 will never be pushed
> to Cluster B again, causing the _last pushed sequenceId_ to stagnate. As a
> result, the {{SerialReplicationChecker}} will repeatedly fail its checks.
> The new RS will keep print DEBUG LOG:
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Replication barrier for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>]:
> ReplicationBarrierResult [barriers=[23, 29, 68], state=OPEN,
> parentRegionNames=]
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Previous range for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>] has not been
> finished yet, give up
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Can not push
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>], wait
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-29463) Bidirectional serial replication will block if a region’s last edit before rs crashed was from the peer cluster
[
https://issues.apache.org/jira/browse/HBASE-29463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18008602#comment-18008602
]
Duo Zhang commented on HBASE-29463:
---
This is a long standing problem for serial replication. Could you please
explain more on why 59-60 will not be replicated again?
> Bidirectional serial replication will block if a region’s last edit before rs
> crashed was from the peer cluster
> ---
>
> Key: HBASE-29463
> URL: https://issues.apache.org/jira/browse/HBASE-29463
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.4.5
>Reporter: haosen chen
>Priority: Major
>
> For two HBase clusters that enable bidirectional replication and set up
> serial replication, when a region in cluster A received last edit from peer
> cluster before RS crashed, the replication from cluster A to B will block.
> Because in this situation, the HBase replication system will wait until the
> last pushed sequence id reaches the new barrier but edit received from peer
> cluster will never be pushed.
> When Region r1 in Cluster A pushes its last edit (e.g., seqID 58) to Cluster
> B and subsequently received two additional edits (seqID 59–60) from Cluster B
> and then the rs crashed, Region r1 will be reopened on another RegionServer
> and set a new barrier at seqID 61. However, edits 59–60 will never be pushed
> to Cluster B again, causing the _last pushed sequenceId_ to stagnate. As a
> result, the {{SerialReplicationChecker}} will repeatedly fail its checks.
> The new RS will keep print DEBUG LOG:
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Replication barrier for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>]:
> ReplicationBarrierResult [barriers=[23, 29, 68], state=OPEN,
> parentRegionNames=]
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Previous range for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>] has not been
> finished yet, give up
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Can not push
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>], wait
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-29463) Bidirectional serial replication will block if a region’s last edit before rs crashed was from the peer cluster
[
https://issues.apache.org/jira/browse/HBASE-29463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18008584#comment-18008584
]
haosen chen commented on HBASE-29463:
-
I think the reason is that the barrier should not be set when the Region is
back online after the RS crashes, because the WAL will also split when the RS
crashes, so there is no problem that the old edit is pushed before the new edit.
> Bidirectional serial replication will block if a region’s last edit before rs
> crashed was from the peer cluster
> ---
>
> Key: HBASE-29463
> URL: https://issues.apache.org/jira/browse/HBASE-29463
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.4.5
>Reporter: haosen chen
>Priority: Major
>
> For two HBase clusters that enable bidirectional replication and set up
> serial replication, when a region in cluster A received last edit from peer
> cluster before RS crashed, the replication from cluster A to B will block.
> Because in this situation, the HBase replication system will wait until the
> last pushed sequence id reaches the new barrier but edit received from peer
> cluster will never be pushed.
> When Region r1 in Cluster A pushes its last edit (e.g., seqID 58) to Cluster
> B and subsequently received two additional edits (seqID 59–60) from Cluster B
> and then the rs crashed, Region r1 will be reopened on another RegionServer
> and set a new barrier at seqID 61. However, edits 59–60 will never be pushed
> to Cluster B again, causing the _last pushed sequenceId_ to stagnate. As a
> result, the {{SerialReplicationChecker}} will repeatedly fail its checks.
> The new RS will keep print DEBUG LOG:
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Replication barrier for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>]:
> ReplicationBarrierResult [barriers=[23, 29, 68], state=OPEN,
> parentRegionNames=]
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Previous range for
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>] has not been
> finished yet, give up
> 2025-07-14 20:05:53,953 DEBUG
> [RS_OPEN_REGION-regionserver/172.16.0.43:6002-0.replicationSource.wal-reader.172.16.0.43%2C6002%2C1752216296629.172.16.0.43%2C6002%2C1752216296629.regiongroup-1,1]
> regionserver.SerialReplicationChecker: Can not push
> test1/46b4ecbd63d7fbcb16d68e106f904013/30=[#edits: 0 = <>], wait
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
