[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[
https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022494#comment-17022494
]
HBase QA commented on HBASE-9888:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 4m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
7s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 4m
25s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 3m
33s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
1s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 1s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 4m
27s{color} | {color:red} patch has 44 errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m
23s{color} | {color:red} The patch causes 44 errors with Hadoop v2.8.5. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m
5s{color} | {color:red} The patch causes 44 errors with Hadoop v2.9.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m
55s{color} | {color:red} The patch causes 44 errors with Hadoop v3.1.2. {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
58s{color} | {color:red} hbase-server in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
38s{color} | {color:green} hbase-replication in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 56s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 base:
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[ https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022447#comment-17022447 ] Michael Stack commented on HBASE-9888: -- Important looking bug. Needs updating and landing in master first. For now unscheduled from branch-2/2.3. > HBase replicates edits written before the replication peer is created > - > > Key: HBASE-9888 > URL: https://issues.apache.org/jira/browse/HBASE-9888 > Project: HBase > Issue Type: Bug >Affects Versions: 1.5.0 >Reporter: Dave Latham >Assignee: Pankaj Kumar >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-9888.002.patch, HBASE-9888.003.patch, > HBASE-9888.branch-1.001.patch, HBASE-9888.branch-1.002.patch, > HBASE-9888.branch-2.002.patch, HBASE-9888.branch-2.patch, > HBASE-9888.branch-2.patch, HBASE-9888.patch > > > When creating a new replication peer the ReplicationSourceManager enqueues > the currently open HLog to the ReplicationSource to ship to the destination > cluster. The ReplicationSource starts at the beginning of the HLog and ships > over any pre-existing writes. > A workaround is to roll all the HLogs before enabling replication. > A little background for how it affected us - we were migrating one cluster in > a master-master pair. I.e. transitioning from A <\-> B to B <-> C. After > shutting down writes from A -> B we enabled writes from C -> B. However, > this replicated some earlier writes that were in C's HLogs that had > originated in A. Since we were running a version of HBase before HBASE-7709 > those writes then got caught in a infinite replication cycle and bringing > down region servers OOM because of HBASE-9865. > However, in general, if one wants to manage what data gets replicated, one > wouldn't expect that potentially very old writes would be included when > setting up a new replication link. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[
https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774531#comment-16774531
]
Hadoop QA commented on HBASE-9888:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
12s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 54s{color}
| {color:red} hbase-server generated 8 new + 180 unchanged - 8 fixed = 188
total (was 188) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
6s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 34s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
37s{color} | {color:green} hbase-replication in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}271m 43s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
45s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}313m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.replication.multiwal.TestReplicationEndpointWithMultipleWAL |
| | hadoop.hbase.master.procedure.TestServerCrashProcedure |
| | hadoop.hbase.master.TestAssignmentManagerMetrics |
| | hadoop.hbase.client.TestFromClientSide3 |
| | hadoop.hbase.replication.TestReplicationEndpoint |
| | hadoop.hbase.replication.TestMultiSlaveReplication |
| |
hadoop.hbase.replication.multiwal.TestReplicationEndpointWithMultipleAsyncWAL |
| | hadoop.hbase.master.procedure.TestServerCrashProcedureWithReplicas |
| | hadoop.hbase.client.TestAdmin1 |
| | hadoop.hbase.client.TestFromClientSide |
\\
\\
|| Subsystem || Report/Notes
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[ https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774289#comment-16774289 ] Pankaj Kumar commented on HBASE-9888: - WALKeyWriteTimeBasedFilter filter is set as an internal filter in 003 patch. > HBase replicates edits written before the replication peer is created > - > > Key: HBASE-9888 > URL: https://issues.apache.org/jira/browse/HBASE-9888 > Project: HBase > Issue Type: Bug >Affects Versions: 1.5.0 >Reporter: Dave Latham >Assignee: Pankaj Kumar >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-9888.002.patch, HBASE-9888.003.patch, > HBASE-9888.branch-1.001.patch, HBASE-9888.branch-1.002.patch, > HBASE-9888.branch-2.002.patch, HBASE-9888.branch-2.patch, > HBASE-9888.branch-2.patch, HBASE-9888.patch > > > When creating a new replication peer the ReplicationSourceManager enqueues > the currently open HLog to the ReplicationSource to ship to the destination > cluster. The ReplicationSource starts at the beginning of the HLog and ships > over any pre-existing writes. > A workaround is to roll all the HLogs before enabling replication. > A little background for how it affected us - we were migrating one cluster in > a master-master pair. I.e. transitioning from A <\-> B to B <-> C. After > shutting down writes from A -> B we enabled writes from C -> B. However, > this replicated some earlier writes that were in C's HLogs that had > originated in A. Since we were running a version of HBase before HBASE-7709 > those writes then got caught in a infinite replication cycle and bringing > down region servers OOM because of HBASE-9865. > However, in general, if one wants to manage what data gets replicated, one > wouldn't expect that potentially very old writes would be included when > setting up a new replication link. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[ https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16730425#comment-16730425 ] Pankaj Kumar commented on HBASE-9888: - We can make WALKeyWriteTimeBasedFilter internal like SystemTableWALEntryFilter, no need to make it configurable. [~stack] sir, please provide your opinion. > HBase replicates edits written before the replication peer is created > - > > Key: HBASE-9888 > URL: https://issues.apache.org/jira/browse/HBASE-9888 > Project: HBase > Issue Type: Bug >Affects Versions: 1.5.0 >Reporter: Dave Latham >Assignee: Pankaj Kumar >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-9888.002.patch, HBASE-9888.branch-1.001.patch, > HBASE-9888.branch-1.002.patch, HBASE-9888.branch-2.002.patch, > HBASE-9888.branch-2.patch, HBASE-9888.branch-2.patch, HBASE-9888.patch > > > When creating a new replication peer the ReplicationSourceManager enqueues > the currently open HLog to the ReplicationSource to ship to the destination > cluster. The ReplicationSource starts at the beginning of the HLog and ships > over any pre-existing writes. > A workaround is to roll all the HLogs before enabling replication. > A little background for how it affected us - we were migrating one cluster in > a master-master pair. I.e. transitioning from A <\-> B to B <-> C. After > shutting down writes from A -> B we enabled writes from C -> B. However, > this replicated some earlier writes that were in C's HLogs that had > originated in A. Since we were running a version of HBase before HBASE-7709 > those writes then got caught in a infinite replication cycle and bringing > down region servers OOM because of HBASE-9865. > However, in general, if one wants to manage what data gets replicated, one > wouldn't expect that potentially very old writes would be included when > setting up a new replication link. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[
https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16728725#comment-16728725
]
Pankaj Kumar commented on HBASE-9888:
-
Thanks Stack Sir for looking into this issue.
{quote}bq. Should this be a filter? It seems like a bug to me? It should be
integral?
{quote}
Yeah it's a bug, should be integral rather than implementing this via filter.
Will check the code and update the patch.
> HBase replicates edits written before the replication peer is created
> -
>
> Key: HBASE-9888
> URL: https://issues.apache.org/jira/browse/HBASE-9888
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.5.0
>Reporter: Dave Latham
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-9888.002.patch, HBASE-9888.branch-1.001.patch,
> HBASE-9888.branch-1.002.patch, HBASE-9888.branch-2.002.patch,
> HBASE-9888.branch-2.patch, HBASE-9888.branch-2.patch, HBASE-9888.patch
>
>
> When creating a new replication peer the ReplicationSourceManager enqueues
> the currently open HLog to the ReplicationSource to ship to the destination
> cluster. The ReplicationSource starts at the beginning of the HLog and ships
> over any pre-existing writes.
> A workaround is to roll all the HLogs before enabling replication.
> A little background for how it affected us - we were migrating one cluster in
> a master-master pair. I.e. transitioning from A <\-> B to B <-> C. After
> shutting down writes from A -> B we enabled writes from C -> B. However,
> this replicated some earlier writes that were in C's HLogs that had
> originated in A. Since we were running a version of HBase before HBASE-7709
> those writes then got caught in a infinite replication cycle and bringing
> down region servers OOM because of HBASE-9865.
> However, in general, if one wants to manage what data gets replicated, one
> wouldn't expect that potentially very old writes would be included when
> setting up a new replication link.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[
https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16728162#comment-16728162
]
Hadoop QA commented on HBASE-9888:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
25s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
52s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
50s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
10s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
25s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 2m
37s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
15s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 15s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
14s{color} | {color:red} hbase-server: The patch generated 2 new + 0 unchanged
- 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 2m
52s{color} | {color:red} patch has 10 errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m
33s{color} | {color:red} The patch causes 10 errors with Hadoop v2.7.4. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m
10s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color}
|
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
13s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
31s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
23s{color} | {color:green} hbase-replication in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 15s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:42ca976 |
| JIRA Issue | HBASE-9888 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12952968/HBASE-9888.branch-2.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 9afdaabfda1d 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[
https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16728051#comment-16728051
]
Hadoop QA commented on HBASE-9888:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
45s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 50s{color}
| {color:red} hbase-server generated 8 new + 180 unchanged - 8 fixed = 188
total (was 188) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
46s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 20s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
44s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
24s{color} | {color:green} hbase-replication in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}127m
12s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}173m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-9888 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12952945/HBASE-9888.002.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 682949bd2a4c 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[ https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16728028#comment-16728028 ] stack commented on HBASE-9888: -- [~pankaj2461] Should this be a filter? It seems like a bug to me? It should be integral? What you think [~pankaj2461]? > HBase replicates edits written before the replication peer is created > - > > Key: HBASE-9888 > URL: https://issues.apache.org/jira/browse/HBASE-9888 > Project: HBase > Issue Type: Bug >Affects Versions: 1.5.0 >Reporter: Dave Latham >Assignee: Pankaj Kumar >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-9888.002.patch, HBASE-9888.branch-1.001.patch, > HBASE-9888.branch-1.002.patch, HBASE-9888.branch-2.002.patch, > HBASE-9888.branch-2.patch, HBASE-9888.patch > > > When creating a new replication peer the ReplicationSourceManager enqueues > the currently open HLog to the ReplicationSource to ship to the destination > cluster. The ReplicationSource starts at the beginning of the HLog and ships > over any pre-existing writes. > A workaround is to roll all the HLogs before enabling replication. > A little background for how it affected us - we were migrating one cluster in > a master-master pair. I.e. transitioning from A <\-> B to B <-> C. After > shutting down writes from A -> B we enabled writes from C -> B. However, > this replicated some earlier writes that were in C's HLogs that had > originated in A. Since we were running a version of HBase before HBASE-7709 > those writes then got caught in a infinite replication cycle and bringing > down region servers OOM because of HBASE-9865. > However, in general, if one wants to manage what data gets replicated, one > wouldn't expect that potentially very old writes would be included when > setting up a new replication link. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[ https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16728009#comment-16728009 ] Pankaj Kumar commented on HBASE-9888: - Raised HBASE-21637 to address TestCheckTestClasses failure. > HBase replicates edits written before the replication peer is created > - > > Key: HBASE-9888 > URL: https://issues.apache.org/jira/browse/HBASE-9888 > Project: HBase > Issue Type: Bug >Affects Versions: 1.5.0 >Reporter: Dave Latham >Assignee: Pankaj Kumar >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-9888.002.patch, HBASE-9888.branch-1.001.patch, > HBASE-9888.branch-1.002.patch, HBASE-9888.branch-2.002.patch, > HBASE-9888.branch-2.patch, HBASE-9888.patch > > > When creating a new replication peer the ReplicationSourceManager enqueues > the currently open HLog to the ReplicationSource to ship to the destination > cluster. The ReplicationSource starts at the beginning of the HLog and ships > over any pre-existing writes. > A workaround is to roll all the HLogs before enabling replication. > A little background for how it affected us - we were migrating one cluster in > a master-master pair. I.e. transitioning from A <\-> B to B <-> C. After > shutting down writes from A -> B we enabled writes from C -> B. However, > this replicated some earlier writes that were in C's HLogs that had > originated in A. Since we were running a version of HBase before HBASE-7709 > those writes then got caught in a infinite replication cycle and bringing > down region servers OOM because of HBASE-9865. > However, in general, if one wants to manage what data gets replicated, one > wouldn't expect that potentially very old writes would be included when > setting up a new replication link. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[ https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16728006#comment-16728006 ] Pankaj Kumar commented on HBASE-9888: - Addressed the checkstyle bug in .002 patch. ping [~anoop.hbase] [~Apache9], kindly review. > HBase replicates edits written before the replication peer is created > - > > Key: HBASE-9888 > URL: https://issues.apache.org/jira/browse/HBASE-9888 > Project: HBase > Issue Type: Bug >Affects Versions: 1.5.0 >Reporter: Dave Latham >Assignee: Pankaj Kumar >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-9888.002.patch, HBASE-9888.branch-1.001.patch, > HBASE-9888.branch-1.002.patch, HBASE-9888.branch-2.002.patch, > HBASE-9888.branch-2.patch, HBASE-9888.patch > > > When creating a new replication peer the ReplicationSourceManager enqueues > the currently open HLog to the ReplicationSource to ship to the destination > cluster. The ReplicationSource starts at the beginning of the HLog and ships > over any pre-existing writes. > A workaround is to roll all the HLogs before enabling replication. > A little background for how it affected us - we were migrating one cluster in > a master-master pair. I.e. transitioning from A <\-> B to B <-> C. After > shutting down writes from A -> B we enabled writes from C -> B. However, > this replicated some earlier writes that were in C's HLogs that had > originated in A. Since we were running a version of HBase before HBASE-7709 > those writes then got caught in a infinite replication cycle and bringing > down region servers OOM because of HBASE-9865. > However, in general, if one wants to manage what data gets replicated, one > wouldn't expect that potentially very old writes would be included when > setting up a new replication link. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[
https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727486#comment-16727486
]
Hadoop QA commented on HBASE-9888:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
57s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
30s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
39s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
26s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
3s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 44s{color}
| {color:red} hbase-server generated 8 new + 180 unchanged - 8 fixed = 188
total (was 188) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
1s{color} | {color:red} hbase-server: The patch generated 2 new + 0 unchanged -
0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
24s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 2s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
42s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
23s{color} | {color:green} hbase-replication in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}114m 14s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}159m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.TestCheckTestClasses |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:42ca976 |
| JIRA Issue | HBASE-9888 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12952907/HBASE-9888.branch-2.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux fd945a4a9b2d 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[ https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727387#comment-16727387 ] Pankaj Kumar commented on HBASE-9888: - Attached patch for master and branch-2, kindly review. > HBase replicates edits written before the replication peer is created > - > > Key: HBASE-9888 > URL: https://issues.apache.org/jira/browse/HBASE-9888 > Project: HBase > Issue Type: Bug >Affects Versions: 1.5.0 >Reporter: Dave Latham >Assignee: Pankaj Kumar >Priority: Major > Attachments: HBASE-9888.branch-1.001.patch, > HBASE-9888.branch-1.002.patch, HBASE-9888.branch-2.patch, HBASE-9888.patch > > > When creating a new replication peer the ReplicationSourceManager enqueues > the currently open HLog to the ReplicationSource to ship to the destination > cluster. The ReplicationSource starts at the beginning of the HLog and ships > over any pre-existing writes. > A workaround is to roll all the HLogs before enabling replication. > A little background for how it affected us - we were migrating one cluster in > a master-master pair. I.e. transitioning from A <\-> B to B <-> C. After > shutting down writes from A -> B we enabled writes from C -> B. However, > this replicated some earlier writes that were in C's HLogs that had > originated in A. Since we were running a version of HBase before HBASE-7709 > those writes then got caught in a infinite replication cycle and bringing > down region servers OOM because of HBASE-9865. > However, in general, if one wants to manage what data gets replicated, one > wouldn't expect that potentially very old writes would be included when > setting up a new replication link. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[ https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555225#comment-16555225 ] Pankaj Kumar commented on HBASE-9888: - Reattached 002 patch for QA run. > HBase replicates edits written before the replication peer is created > - > > Key: HBASE-9888 > URL: https://issues.apache.org/jira/browse/HBASE-9888 > Project: HBase > Issue Type: Bug >Affects Versions: 1.5.0 >Reporter: Dave Latham >Priority: Major > Attachments: HBASE-9888.branch-1.001.patch, > HBASE-9888.branch-1.002.patch > > > When creating a new replication peer the ReplicationSourceManager enqueues > the currently open HLog to the ReplicationSource to ship to the destination > cluster. The ReplicationSource starts at the beginning of the HLog and ships > over any pre-existing writes. > A workaround is to roll all the HLogs before enabling replication. > A little background for how it affected us - we were migrating one cluster in > a master-master pair. I.e. transitioning from A <\-> B to B <-> C. After > shutting down writes from A -> B we enabled writes from C -> B. However, > this replicated some earlier writes that were in C's HLogs that had > originated in A. Since we were running a version of HBase before HBASE-7709 > those writes then got caught in a infinite replication cycle and bringing > down region servers OOM because of HBASE-9865. > However, in general, if one wants to manage what data gets replicated, one > wouldn't expect that potentially very old writes would be included when > setting up a new replication link. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[
https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554723#comment-16554723
]
Pankaj Kumar commented on HBASE-9888:
-
Thanks [~yuzhih...@gmail.com] and [~xucang] for reviewing the patch. Have
addressed them in .002 patch.
{quote}In TestReplicationEndpoint.java, the new flag is set to false. Is there
any test failing in this class ?
{quote}
Yes, TestReplicationEndpoint.testInterClusterReplication() and
TestMultiSlaveReplication.testMultiSlaveReplication() are failing.
{quote}For master branch, can similar solution be applied ?
{quote}
I just tried with the test case, it fails there also. Will check the solution
approach and attach the patch.
> HBase replicates edits written before the replication peer is created
> -
>
> Key: HBASE-9888
> URL: https://issues.apache.org/jira/browse/HBASE-9888
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.5.0
>Reporter: Dave Latham
>Priority: Major
> Attachments: HBASE-9888.branch-1.001.patch,
> HBASE-9888.branch-1.002.patch
>
>
> When creating a new replication peer the ReplicationSourceManager enqueues
> the currently open HLog to the ReplicationSource to ship to the destination
> cluster. The ReplicationSource starts at the beginning of the HLog and ships
> over any pre-existing writes.
> A workaround is to roll all the HLogs before enabling replication.
> A little background for how it affected us - we were migrating one cluster in
> a master-master pair. I.e. transitioning from A <\-> B to B <-> C. After
> shutting down writes from A -> B we enabled writes from C -> B. However,
> this replicated some earlier writes that were in C's HLogs that had
> originated in A. Since we were running a version of HBase before HBASE-7709
> those writes then got caught in a infinite replication cycle and bringing
> down region servers OOM because of HBASE-9865.
> However, in general, if one wants to manage what data gets replicated, one
> wouldn't expect that potentially very old writes would be included when
> setting up a new replication link.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[
https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554511#comment-16554511
]
Hadoop QA commented on HBASE-9888:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
10s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
37s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
24s{color} | {color:red} hbase-common: The patch generated 1 new + 9 unchanged
- 1 fixed = 10 total (was 10) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
29s{color} | {color:red} hbase-client: The patch generated 3 new + 18 unchanged
- 0 fixed = 21 total (was 18) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
37s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 34s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
9s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
21s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 36s{color}
| {color:red} hbase-server in the patch failed. {color} |
|
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[
https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554487#comment-16554487
]
Xu Cang commented on HBASE-9888:
{code:java}
conf1.setBoolean("hbase.replication.walKeyWriteTime.filter.enabled",
false);{code}
I see you only test the case that filter.enabled is false. Where is the test
when "filter.enabled" is true? Sorry if I am misreading it. thanks.
{code:java}
+ assertTrue("table data should be deleted from source",
utility1.countRows(htable1) == 1);
{code}
> HBase replicates edits written before the replication peer is created
> -
>
> Key: HBASE-9888
> URL: https://issues.apache.org/jira/browse/HBASE-9888
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.5.0
>Reporter: Dave Latham
>Priority: Major
> Attachments: HBASE-9888.branch-1.001.patch
>
>
> When creating a new replication peer the ReplicationSourceManager enqueues
> the currently open HLog to the ReplicationSource to ship to the destination
> cluster. The ReplicationSource starts at the beginning of the HLog and ships
> over any pre-existing writes.
> A workaround is to roll all the HLogs before enabling replication.
> A little background for how it affected us - we were migrating one cluster in
> a master-master pair. I.e. transitioning from A <\-> B to B <-> C. After
> shutting down writes from A -> B we enabled writes from C -> B. However,
> this replicated some earlier writes that were in C's HLogs that had
> originated in A. Since we were running a version of HBase before HBASE-7709
> those writes then got caught in a infinite replication cycle and bringing
> down region servers OOM because of HBASE-9865.
> However, in general, if one wants to manage what data gets replicated, one
> wouldn't expect that potentially very old writes would be included when
> setting up a new replication link.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[
https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554352#comment-16554352
]
Ted Yu commented on HBASE-9888:
---
{code}
+ * @return create time for a replication peer
{code}
create time -> creation time
{code}
+ private long createTime = Long.MIN_VALUE;
{code}
createTime -> creationTime
{code}
+ void initCreateTime(final ZooKeeperWatcher zookeeper, String peerNode)
throws KeeperException {
{code}
initCreateTime -> initCreationTime
In TestReplicationEndpoint.java, the new flag is set to false. Is there any
test failing in this class ?
For master branch, can similar solution be applied ?
> HBase replicates edits written before the replication peer is created
> -
>
> Key: HBASE-9888
> URL: https://issues.apache.org/jira/browse/HBASE-9888
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.5.0
>Reporter: Dave Latham
>Priority: Major
> Attachments: HBASE-9888.branch-1.001.patch
>
>
> When creating a new replication peer the ReplicationSourceManager enqueues
> the currently open HLog to the ReplicationSource to ship to the destination
> cluster. The ReplicationSource starts at the beginning of the HLog and ships
> over any pre-existing writes.
> A workaround is to roll all the HLogs before enabling replication.
> A little background for how it affected us - we were migrating one cluster in
> a master-master pair. I.e. transitioning from A <\-> B to B <-> C. After
> shutting down writes from A -> B we enabled writes from C -> B. However,
> this replicated some earlier writes that were in C's HLogs that had
> originated in A. Since we were running a version of HBase before HBASE-7709
> those writes then got caught in a infinite replication cycle and bringing
> down region servers OOM because of HBASE-9865.
> However, in general, if one wants to manage what data gets replicated, one
> wouldn't expect that potentially very old writes would be included when
> setting up a new replication link.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[ https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554290#comment-16554290 ] Pankaj Kumar commented on HBASE-9888: - [~anoop.hbase] [~davelatham] ... Please review. > HBase replicates edits written before the replication peer is created > - > > Key: HBASE-9888 > URL: https://issues.apache.org/jira/browse/HBASE-9888 > Project: HBase > Issue Type: Bug >Affects Versions: 1.5.0 >Reporter: Dave Latham >Priority: Major > Attachments: HBASE-9888.branch-1.001.patch > > > When creating a new replication peer the ReplicationSourceManager enqueues > the currently open HLog to the ReplicationSource to ship to the destination > cluster. The ReplicationSource starts at the beginning of the HLog and ships > over any pre-existing writes. > A workaround is to roll all the HLogs before enabling replication. > A little background for how it affected us - we were migrating one cluster in > a master-master pair. I.e. transitioning from A <\-> B to B <-> C. After > shutting down writes from A -> B we enabled writes from C -> B. However, > this replicated some earlier writes that were in C's HLogs that had > originated in A. Since we were running a version of HBase before HBASE-7709 > those writes then got caught in a infinite replication cycle and bringing > down region servers OOM because of HBASE-9865. > However, in general, if one wants to manage what data gets replicated, one > wouldn't expect that potentially very old writes would be included when > setting up a new replication link. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[ https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16539666#comment-16539666 ] Pankaj Kumar commented on HBASE-9888: - Recently we met this problem in our production environment, have fixed this problem using filter. Let me know if nobody working on this, I will take this. > HBase replicates edits written before the replication peer is created > - > > Key: HBASE-9888 > URL: https://issues.apache.org/jira/browse/HBASE-9888 > Project: HBase > Issue Type: Bug >Reporter: Dave Latham >Priority: Major > > When creating a new replication peer the ReplicationSourceManager enqueues > the currently open HLog to the ReplicationSource to ship to the destination > cluster. The ReplicationSource starts at the beginning of the HLog and ships > over any pre-existing writes. > A workaround is to roll all the HLogs before enabling replication. > A little background for how it affected us - we were migrating one cluster in > a master-master pair. I.e. transitioning from A <\-> B to B <-> C. After > shutting down writes from A -> B we enabled writes from C -> B. However, > this replicated some earlier writes that were in C's HLogs that had > originated in A. Since we were running a version of HBase before HBASE-7709 > those writes then got caught in a infinite replication cycle and bringing > down region servers OOM because of HBASE-9865. > However, in general, if one wants to manage what data gets replicated, one > wouldn't expect that potentially very old writes would be included when > setting up a new replication link. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[
https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13815084#comment-13815084
]
Jean-Daniel Cryans commented on HBASE-9888:
---
In 0.94, HLogKey has a {{writeTime}} and we could seek in the current WAL until
we find an edit that's been written after the source was created. It's still
fuzzy since the time that each source actually gets created will differ for
each RS, but at least you wouldn't start replicating old edits.
HBase replicates edits written before the replication peer is created
-
Key: HBASE-9888
URL: https://issues.apache.org/jira/browse/HBASE-9888
Project: HBase
Issue Type: Bug
Reporter: Dave Latham
When creating a new replication peer the ReplicationSourceManager enqueues
the currently open HLog to the ReplicationSource to ship to the destination
cluster. The ReplicationSource starts at the beginning of the HLog and ships
over any pre-existing writes.
A workaround is to roll all the HLogs before enabling replication.
A little background for how it affected us - we were migrating one cluster in
a master-master pair. I.e. transitioning from A \- B to B - C. After
shutting down writes from A - B we enabled writes from C - B. However,
this replicated some earlier writes that were in C's HLogs that had
originated in A. Since we were running a version of HBase before HBASE-7709
those writes then got caught in a infinite replication cycle and bringing
down region servers OOM because of HBASE-9865.
However, in general, if one wants to manage what data gets replicated, one
wouldn't expect that potentially very old writes would be included when
setting up a new replication link.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[
https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13815111#comment-13815111
]
Dave Latham commented on HBASE-9888:
{quote}
So in your case the replication was enabled at the master cluster and the table
cf was also replication enabled (scope0) At a later point u added a peer to
the master and you can see the older edits also getting replicated. The
scenario is correct?
{quote}
Yes, that's correct.
{quote}
Did you consider writing a replication controller to be deployed on the sink
cluster ?
{quote}
Nope, didn't consider it because we weren't aware of the issue ahead of time.
We ended up resolving the infinite replication cycle by introducing a patch to
allow configuring specific cluster UUIDs whose edits should not be replication.
For the future, I'd prefer a built in improvement.
{quote}
In 0.94, HLogKey has a writeTime and we could seek in the current WAL until we
find an edit that's been written after the source was created. It's still fuzzy
since the time that each source actually gets created will differ for each RS,
but at least you wouldn't start replicating old edits.
{quote}
That sounds great. Is that 0.94 only or do the newer versions also have it?
Do you have an idea where the minimum timestamp would be generated? Would it
work to just do it in each RS when the ReplicationSource on that RS is created
(in the mode for add_peer)?
Alternatively, should each RS roll its HLog when creating a new peer?
HBase replicates edits written before the replication peer is created
-
Key: HBASE-9888
URL: https://issues.apache.org/jira/browse/HBASE-9888
Project: HBase
Issue Type: Bug
Reporter: Dave Latham
When creating a new replication peer the ReplicationSourceManager enqueues
the currently open HLog to the ReplicationSource to ship to the destination
cluster. The ReplicationSource starts at the beginning of the HLog and ships
over any pre-existing writes.
A workaround is to roll all the HLogs before enabling replication.
A little background for how it affected us - we were migrating one cluster in
a master-master pair. I.e. transitioning from A \- B to B - C. After
shutting down writes from A - B we enabled writes from C - B. However,
this replicated some earlier writes that were in C's HLogs that had
originated in A. Since we were running a version of HBase before HBASE-7709
those writes then got caught in a infinite replication cycle and bringing
down region servers OOM because of HBASE-9865.
However, in general, if one wants to manage what data gets replicated, one
wouldn't expect that potentially very old writes would be included when
setting up a new replication link.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[
https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13815313#comment-13815313
]
santosh banerjee commented on HBASE-9888:
-
{quote} In 0.94, HLogKey has a writeTime and we could seek in the current WAL
until we find an edit that's been written after the source was created.{quote}
This sounds interesting. So,are you suggesting implementing a custom
ReplicationSource to seek into the current WAL to find the edit with writeTime
sourceCreationTimestamp?
HBase replicates edits written before the replication peer is created
-
Key: HBASE-9888
URL: https://issues.apache.org/jira/browse/HBASE-9888
Project: HBase
Issue Type: Bug
Reporter: Dave Latham
When creating a new replication peer the ReplicationSourceManager enqueues
the currently open HLog to the ReplicationSource to ship to the destination
cluster. The ReplicationSource starts at the beginning of the HLog and ships
over any pre-existing writes.
A workaround is to roll all the HLogs before enabling replication.
A little background for how it affected us - we were migrating one cluster in
a master-master pair. I.e. transitioning from A \- B to B - C. After
shutting down writes from A - B we enabled writes from C - B. However,
this replicated some earlier writes that were in C's HLogs that had
originated in A. Since we were running a version of HBase before HBASE-7709
those writes then got caught in a infinite replication cycle and bringing
down region servers OOM because of HBASE-9865.
However, in general, if one wants to manage what data gets replicated, one
wouldn't expect that potentially very old writes would be included when
setting up a new replication link.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[ https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13815314#comment-13815314 ] Jean-Daniel Cryans commented on HBASE-9888: --- bq. That sounds great. Is that 0.94 only or do the newer versions also have it? It's in trunk too. bq. Do you have an idea where the minimum timestamp would be generated? Once we get the zk event? Not sure. bq. Would it work to just do it in each RS when the ReplicationSource on that RS is created (in the mode for add_peer)? That's what I was proposing, sorry if not clear. bq. Alternatively, should each RS roll its HLog when creating a new peer? That could work but I'd rather not roll logs for this. HBase replicates edits written before the replication peer is created - Key: HBASE-9888 URL: https://issues.apache.org/jira/browse/HBASE-9888 Project: HBase Issue Type: Bug Reporter: Dave Latham When creating a new replication peer the ReplicationSourceManager enqueues the currently open HLog to the ReplicationSource to ship to the destination cluster. The ReplicationSource starts at the beginning of the HLog and ships over any pre-existing writes. A workaround is to roll all the HLogs before enabling replication. A little background for how it affected us - we were migrating one cluster in a master-master pair. I.e. transitioning from A \- B to B - C. After shutting down writes from A - B we enabled writes from C - B. However, this replicated some earlier writes that were in C's HLogs that had originated in A. Since we were running a version of HBase before HBASE-7709 those writes then got caught in a infinite replication cycle and bringing down region servers OOM because of HBASE-9865. However, in general, if one wants to manage what data gets replicated, one wouldn't expect that potentially very old writes would be included when setting up a new replication link. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[ https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13815322#comment-13815322 ] Jean-Daniel Cryans commented on HBASE-9888: --- bq. So,are you suggesting implementing a custom ReplicationSource to seek into the current WAL to find the edit with writeTime sourceCreationTimestamp? It wouldn't be custom, it'd be the default behavior. HBase replicates edits written before the replication peer is created - Key: HBASE-9888 URL: https://issues.apache.org/jira/browse/HBASE-9888 Project: HBase Issue Type: Bug Reporter: Dave Latham When creating a new replication peer the ReplicationSourceManager enqueues the currently open HLog to the ReplicationSource to ship to the destination cluster. The ReplicationSource starts at the beginning of the HLog and ships over any pre-existing writes. A workaround is to roll all the HLogs before enabling replication. A little background for how it affected us - we were migrating one cluster in a master-master pair. I.e. transitioning from A \- B to B - C. After shutting down writes from A - B we enabled writes from C - B. However, this replicated some earlier writes that were in C's HLogs that had originated in A. Since we were running a version of HBase before HBASE-7709 those writes then got caught in a infinite replication cycle and bringing down region servers OOM because of HBASE-9865. However, in general, if one wants to manage what data gets replicated, one wouldn't expect that potentially very old writes would be included when setting up a new replication link. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[
https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13815337#comment-13815337
]
Dave Latham commented on HBASE-9888:
{quote}
Would it work to just do it in each RS when the ReplicationSource on that RS
is created (in the mode for add_peer)?
That's what I was proposing, sorry if not clear.
{quote}
+1 for the proposal.
HBase replicates edits written before the replication peer is created
-
Key: HBASE-9888
URL: https://issues.apache.org/jira/browse/HBASE-9888
Project: HBase
Issue Type: Bug
Reporter: Dave Latham
When creating a new replication peer the ReplicationSourceManager enqueues
the currently open HLog to the ReplicationSource to ship to the destination
cluster. The ReplicationSource starts at the beginning of the HLog and ships
over any pre-existing writes.
A workaround is to roll all the HLogs before enabling replication.
A little background for how it affected us - we were migrating one cluster in
a master-master pair. I.e. transitioning from A \- B to B - C. After
shutting down writes from A - B we enabled writes from C - B. However,
this replicated some earlier writes that were in C's HLogs that had
originated in A. Since we were running a version of HBase before HBASE-7709
those writes then got caught in a infinite replication cycle and bringing
down region servers OOM because of HBASE-9865.
However, in general, if one wants to manage what data gets replicated, one
wouldn't expect that potentially very old writes would be included when
setting up a new replication link.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[ https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13814686#comment-13814686 ] santosh banerjee commented on HBASE-9888: - Did you consider writing a replication controller to be deployed on the sink cluster ? The idea is to implement a edits filter plugin as a ReplicationSinkService implementation that can be defined in the hbase-site.xml as follows? property namehbase.replication.source.service/name valuecom..replication.ReplicationController/value /property You can consider extending the default implementation org.apache.hadoop.hbase.replication.regionserver.Replication of ReplicationSinkService, and just override the replicateLogEntries() method to apply your filtering logic. For your specific usecase, your sink cluster needs to know when it was added as a peer, and then ,assuming the two clusters are time-synced, the custom sink service can filter the edits based on this timestamp, leaving out the ones that were created earlier than the timestamp. HBase replicates edits written before the replication peer is created - Key: HBASE-9888 URL: https://issues.apache.org/jira/browse/HBASE-9888 Project: HBase Issue Type: Bug Reporter: Dave Latham When creating a new replication peer the ReplicationSourceManager enqueues the currently open HLog to the ReplicationSource to ship to the destination cluster. The ReplicationSource starts at the beginning of the HLog and ships over any pre-existing writes. A workaround is to roll all the HLogs before enabling replication. A little background for how it affected us - we were migrating one cluster in a master-master pair. I.e. transitioning from A \- B to B - C. After shutting down writes from A - B we enabled writes from C - B. However, this replicated some earlier writes that were in C's HLogs that had originated in A. Since we were running a version of HBase before HBASE-7709 those writes then got caught in a infinite replication cycle and bringing down region servers OOM because of HBASE-9865. However, in general, if one wants to manage what data gets replicated, one wouldn't expect that potentially very old writes would be included when setting up a new replication link. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9888) HBase replicates edits written before the replication peer is created
[ https://issues.apache.org/jira/browse/HBASE-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13813663#comment-13813663 ] Anoop Sam John commented on HBASE-9888: --- So in your case the replication was enabled at the master cluster and the table cf was also replication enabled (scope0) At a later point u added a peer to the master and you can see the older edits also getting replicated. The scenario is correct? HBase replicates edits written before the replication peer is created - Key: HBASE-9888 URL: https://issues.apache.org/jira/browse/HBASE-9888 Project: HBase Issue Type: Bug Reporter: Dave Latham When creating a new replication peer the ReplicationSourceManager enqueues the currently open HLog to the ReplicationSource to ship to the destination cluster. The ReplicationSource starts at the beginning of the HLog and ships over any pre-existing writes. A workaround is to roll all the HLogs before enabling replication. A little background for how it affected us - we were migrating one cluster in a master-master pair. I.e. transitioning from A - B to B - C. After shutting down writes from A - B we enabled writes from C - B. However, this replicated some earlier writes that were in C's HLogs that had originated in A. Since we were running a version of HBase before HBASE-7709 those writes then got caught in a infinite replication cycle and bringing down region servers OOM because of HBASE-9865. However, in general, if one wants to manage what data gets replicated, one wouldn't expect that potentially very old writes would be included when setting up a new replication link. -- This message was sent by Atlassian JIRA (v6.1#6144)
