[jira] [Commented] (HBASE-20456) Support removing a ReplicationSourceShipper for a special wal group
[ https://issues.apache.org/jira/browse/HBASE-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693904#comment-16693904 ] Wellington Chevreuil commented on HBASE-20456: -- Actually, this change has just allowed issue described on HBASE-21503 to surface, it's not really the cause. I guess no extra work is needed here. > Support removing a ReplicationSourceShipper for a special wal group > --- > > Key: HBASE-20456 > URL: https://issues.apache.org/jira/browse/HBASE-20456 > Project: HBase > Issue Type: Sub-task > Components: Replication, wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20456-HBASE-19064-v1.patch, > HBASE-20456-HBASE-19064-v2.patch, HBASE-20456-HBASE-19064-v3.patch, > HBASE-20456-HBASE-19064-v3.patch, HBASE-20456-HBASE-19064.patch > > > For the multi wal implementation, if a new group is created, then we will > always open a wal writer for it, even if no one uses it later. But for sync > replication, if the peer is transited to DA or S from A, the special wal will > be closed and there will be no more wal files for it so we need to close the > shipper -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20456) Support removing a ReplicationSourceShipper for a special wal group
[
https://issues.apache.org/jira/browse/HBASE-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689696#comment-16689696
]
Wellington Chevreuil commented on HBASE-20456:
--
Hi [~Apache9],
Was doing some tests with replication on master branch, and I believe there's
an issue caused by the following change introduced by this patch:
{noformat}
- private void handleEmptyWALEntryBatch(Path currentPath) throws
InterruptedException {
+ private void handleEmptyWALEntryBatch() throws InterruptedException {
LOG.trace("Didn't read any new entries from WAL");
-if (source.isRecovered()) {
- // we're done with queue recovery, shut ourself down
+if (logQueue.isEmpty()) {
+ // we're done with current queue, either this is a recovered queue, or
it is the special group
+ // for a sync replication peer and the peer has been transited to DA or
S state.
{noformat}
The problem is that for a normal source (not replicated), if it starts while
the current WAL is still empty (for instance, just after an RS restart), the
reader will reach WAL's end, then shutdown the source. If then new edits are
added to this same WAL, it won't get replicated until this wal file gets into a
recovery queue.
This is easy to reproduce, just setup replication, restart source RS, do a put
on source, related edit does not get into target until the wal goes to a
recovery queue (restarting RS, for instance, causes the wal containing that put
to be recovered).
> Support removing a ReplicationSourceShipper for a special wal group
> ---
>
> Key: HBASE-20456
> URL: https://issues.apache.org/jira/browse/HBASE-20456
> Project: HBase
> Issue Type: Sub-task
> Components: Replication, wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20456-HBASE-19064-v1.patch,
> HBASE-20456-HBASE-19064-v2.patch, HBASE-20456-HBASE-19064-v3.patch,
> HBASE-20456-HBASE-19064-v3.patch, HBASE-20456-HBASE-19064.patch
>
>
> For the multi wal implementation, if a new group is created, then we will
> always open a wal writer for it, even if no one uses it later. But for sync
> replication, if the peer is transited to DA or S from A, the special wal will
> be closed and there will be no more wal files for it so we need to close the
> shipper
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20456) Support removing a ReplicationSourceShipper for a special wal group
[
https://issues.apache.org/jira/browse/HBASE-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527707#comment-16527707
]
Hudson commented on HBASE-20456:
Results for branch master
[build #380 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/380/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/380//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/380//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/380//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Support removing a ReplicationSourceShipper for a special wal group
> ---
>
> Key: HBASE-20456
> URL: https://issues.apache.org/jira/browse/HBASE-20456
> Project: HBase
> Issue Type: Sub-task
> Components: Replication, wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20456-HBASE-19064-v1.patch,
> HBASE-20456-HBASE-19064-v2.patch, HBASE-20456-HBASE-19064-v3.patch,
> HBASE-20456-HBASE-19064-v3.patch, HBASE-20456-HBASE-19064.patch
>
>
> For the multi wal implementation, if a new group is created, then we will
> always open a wal writer for it, even if no one uses it later. But for sync
> replication, if the peer is transited to DA or S from A, the special wal will
> be closed and there will be no more wal files for it so we need to close the
> shipper
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20456) Support removing a ReplicationSourceShipper for a special wal group
[
https://issues.apache.org/jira/browse/HBASE-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449220#comment-16449220
]
Hadoop QA commented on HBASE-20456:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} HBASE-19064 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
37s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
53s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} HBASE-19064 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 43s{color}
| {color:red} hbase-server generated 1 new + 187 unchanged - 1 fixed = 188
total (was 188) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
48s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 38s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}106m
32s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}152m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20456 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12920379/HBASE-20456-HBASE-19064-v3.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 2403216247c4 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | HBASE-19064 / afd78256c5 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC3 |
| javac |
https://builds.apache.org/job/PreCommit-HBASE-Build/12607/artifact/patchprocess/diff-compile-javac-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/12607/testReport/ |
| Max. process+thread count | 4265 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
|
[jira] [Commented] (HBASE-20456) Support removing a ReplicationSourceShipper for a special wal group
[ https://issues.apache.org/jira/browse/HBASE-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449114#comment-16449114 ] Duo Zhang commented on HBASE-20456: --- Retry. The javac warning is not introduced by this patch. And the failed UTs are not related I suppose. > Support removing a ReplicationSourceShipper for a special wal group > --- > > Key: HBASE-20456 > URL: https://issues.apache.org/jira/browse/HBASE-20456 > Project: HBase > Issue Type: Sub-task > Components: Replication, wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: HBASE-19064 > > Attachments: HBASE-20456-HBASE-19064-v1.patch, > HBASE-20456-HBASE-19064-v2.patch, HBASE-20456-HBASE-19064-v3.patch, > HBASE-20456-HBASE-19064-v3.patch, HBASE-20456-HBASE-19064.patch > > > For the multi wal implementation, if a new group is created, then we will > always open a wal writer for it, even if no one uses it later. But for sync > replication, if the peer is transited to DA or S from A, the special wal will > be closed and there will be no more wal files for it so we need to close the > shipper -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20456) Support removing a ReplicationSourceShipper for a special wal group
[
https://issues.apache.org/jira/browse/HBASE-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16448425#comment-16448425
]
Hadoop QA commented on HBASE-20456:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} HBASE-19064 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
55s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
1s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
18s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} HBASE-19064 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 48s{color}
| {color:red} hbase-server generated 1 new + 187 unchanged - 1 fixed = 188
total (was 188) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
55s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
16m 31s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}146m 46s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}197m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.procedure.TestModifyNamespaceProcedure |
| |
hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaHighReplication
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20456 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12920283/HBASE-20456-HBASE-19064-v3.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux dbe7646df1da 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | HBASE-19064 / afd78256c5 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC3 |
| javac |
https://builds.apache.org/job/PreCommit-HBASE-Build/12603/artifact/patchprocess/diff-compile-javac-hbase-server.txt
|
[jira] [Commented] (HBASE-20456) Support removing a ReplicationSourceShipper for a special wal group
[
https://issues.apache.org/jira/browse/HBASE-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16448132#comment-16448132
]
Hadoop QA commented on HBASE-20456:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HBASE-19064 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
37s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
39s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} HBASE-19064 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 47s{color}
| {color:red} hbase-server generated 1 new + 187 unchanged - 1 fixed = 188
total (was 188) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
9s{color} | {color:red} hbase-server: The patch generated 1 new + 10 unchanged
- 0 fixed = 11 total (was 10) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
46s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
13m 8s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}114m 39s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}158m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.replication.regionserver.TestWALEntryStream
|
| | hadoop.hbase.replication.regionserver.TestReplicationSourceManagerZkImpl |
| | hadoop.hbase.util.TestMiniClusterLoadParallel |
| | hadoop.hbase.util.TestHBaseFsckReplication |
| | hadoop.hbase.TestCheckTestClasses |
| | hadoop.hbase.util.TestRegionSplitter |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20456 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12920265/HBASE-20456-HBASE-19064-v2.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 9e2822d68b7f 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | HBASE-19064 / afd78256c5 |
| maven | version: Apache Maven 3.5.3
[jira] [Commented] (HBASE-20456) Support removing a ReplicationSourceShipper for a special wal group
[ https://issues.apache.org/jira/browse/HBASE-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16448131#comment-16448131 ] Duo Zhang commented on HBASE-20456: --- Discussed with [~zghaobac] again, the difficulty here is that, we assume that in DA state a region will not use the special WAL for sync replication peer, so when removing the peer we need to do nothing. So if we always use a special WAL then after removing the peer, the WAL is still there... And also, the wal files will be added to all replication sources, so even if we remove the peer, replication sources for other peers will still create a shipper for it. Another way maybe use the same group which we use in DA or S state, but we could also use multiwal, then we can not know the group unless we actual get a WAL instance... This is a bit ugly... We get a WAL using the old WALProvider, and then drop it and create our own WAL... > Support removing a ReplicationSourceShipper for a special wal group > --- > > Key: HBASE-20456 > URL: https://issues.apache.org/jira/browse/HBASE-20456 > Project: HBase > Issue Type: Sub-task > Components: Replication, wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: HBASE-19064 > > Attachments: HBASE-20456-HBASE-19064-v1.patch, > HBASE-20456-HBASE-19064-v2.patch, HBASE-20456-HBASE-19064-v3.patch, > HBASE-20456-HBASE-19064.patch > > > For the multi wal implementation, if a new group is created, then we will > always open a wal writer for it, even if no one uses it later. But for sync > replication, if the peer is transited to DA or S from A, the special wal will > be closed and there will be no more wal files for it so we need to close the > shipper -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20456) Support removing a ReplicationSourceShipper for a special wal group
[ https://issues.apache.org/jira/browse/HBASE-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16448118#comment-16448118 ] Duo Zhang commented on HBASE-20456: --- Fix failed UTs. > Support removing a ReplicationSourceShipper for a special wal group > --- > > Key: HBASE-20456 > URL: https://issues.apache.org/jira/browse/HBASE-20456 > Project: HBase > Issue Type: Sub-task > Components: Replication, wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: HBASE-19064 > > Attachments: HBASE-20456-HBASE-19064-v1.patch, > HBASE-20456-HBASE-19064-v2.patch, HBASE-20456-HBASE-19064-v3.patch, > HBASE-20456-HBASE-19064.patch > > > For the multi wal implementation, if a new group is created, then we will > always open a wal writer for it, even if no one uses it later. But for sync > replication, if the peer is transited to DA or S from A, the special wal will > be closed and there will be no more wal files for it so we need to close the > shipper -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20456) Support removing a ReplicationSourceShipper for a special wal group
[
https://issues.apache.org/jira/browse/HBASE-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447969#comment-16447969
]
Hadoop QA commented on HBASE-20456:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HBASE-19064 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
52s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
7s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} HBASE-19064 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 31s{color}
| {color:red} hbase-server generated 1 new + 187 unchanged - 1 fixed = 188
total (was 188) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
57s{color} | {color:red} hbase-server: The patch generated 1 new + 10 unchanged
- 0 fixed = 11 total (was 10) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
10s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
13m 2s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 28s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}150m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.replication.TestSyncReplicationActive |
| | hadoop.hbase.replication.regionserver.TestWALEntryStream |
| | hadoop.hbase.replication.regionserver.TestReplicationSourceManagerZkImpl |
| | hadoop.hbase.TestCheckTestClasses |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20456 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12920245/HBASE-20456-HBASE-19064-v1.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 4d05586fb569 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | HBASE-19064 / afd78256c5 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_162 |
| findbugs
[jira] [Commented] (HBASE-20456) Support removing a ReplicationSourceShipper for a special wal group
[ https://issues.apache.org/jira/browse/HBASE-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447940#comment-16447940 ] Duo Zhang commented on HBASE-20456: --- Talked with [~zghaobac] offline, for peer removal we just remove the ReplicationSource directly so no problem. Let me see if we can do this in the way described above, i.e, always write wal to a special group, even if it is in DA or S. > Support removing a ReplicationSourceShipper for a special wal group > --- > > Key: HBASE-20456 > URL: https://issues.apache.org/jira/browse/HBASE-20456 > Project: HBase > Issue Type: Sub-task > Components: Replication, wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: HBASE-19064 > > Attachments: HBASE-20456-HBASE-19064-v1.patch, > HBASE-20456-HBASE-19064-v2.patch, HBASE-20456-HBASE-19064.patch > > > For the multi wal implementation, if a new group is created, then we will > always open a wal writer for it, even if no one uses it later. But for sync > replication, if the peer is transited to DA or S from A, the special wal will > be closed and there will be no more wal files for it so we need to close the > shipper -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20456) Support removing a ReplicationSourceShipper for a special wal group
[
https://issues.apache.org/jira/browse/HBASE-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447694#comment-16447694
]
Hadoop QA commented on HBASE-20456:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HBASE-20456 does not apply to HBASE-19064. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.7.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-20456 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12920241/HBASE-20456-HBASE-19064.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/12595/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Support removing a ReplicationSourceShipper for a special wal group
> ---
>
> Key: HBASE-20456
> URL: https://issues.apache.org/jira/browse/HBASE-20456
> Project: HBase
> Issue Type: Sub-task
> Components: Replication, wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: HBASE-19064
>
> Attachments: HBASE-20456-HBASE-19064.patch
>
>
> For the multi wal implementation, if a new group is created, then we will
> always open a wal writer for it, even if no one uses it later. But for sync
> replication, if the peer is transited to DA or S from A, the special wal will
> be closed and there will be no more wal files for it so we need to close the
> shipper
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20456) Support removing a ReplicationSourceShipper for a special wal group
[ https://issues.apache.org/jira/browse/HBASE-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447692#comment-16447692 ] Duo Zhang commented on HBASE-20456: --- Review board link: https://reviews.apache.org/r/66756/ > Support removing a ReplicationSourceShipper for a special wal group > --- > > Key: HBASE-20456 > URL: https://issues.apache.org/jira/browse/HBASE-20456 > Project: HBase > Issue Type: Sub-task > Components: Replication, wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: HBASE-19064 > > Attachments: HBASE-20456-HBASE-19064.patch > > > For the multi wal implementation, if a new group is created, then we will > always open a wal writer for it, even if no one uses it later. But for sync > replication, if the peer is transited to DA or S from A, the special wal will > be closed and there will be no more wal files for it so we need to close the > shipper -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20456) Support removing a ReplicationSourceShipper for a special wal group
[ https://issues.apache.org/jira/browse/HBASE-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443618#comment-16443618 ] Duo Zhang commented on HBASE-20456: --- Another way maybe we always create a separated wal group, even if it is not in A state, the only difference is that we will also write to the remote cluster when in A state. But it will be hard to deal with peer removal... And also another places we need to deal with is the LogRoller... It only has a addWAL method... > Support removing a ReplicationSourceShipper for a special wal group > --- > > Key: HBASE-20456 > URL: https://issues.apache.org/jira/browse/HBASE-20456 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Priority: Major > > For the multi wal implementation, if a new group is created, then we will > always open a wal writer for it, even if no one uses it later. But for sync > replication, if the peer is transited to DA or S from A, the special wal will > be closed and there will be no more wal files for it so we need to close the > shipper -- This message was sent by Atlassian JIRA (v7.6.3#76005)
