[jira] [Commented] (HBASE-17825) Backup: further optimizations
[ https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381115#comment-16381115 ] Hudson commented on HBASE-17825: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4666 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4666/]) HBASE-17825: Backup further optimizations (elserj: rev 6cfa208add6ea424e17cae00114ebd3e7d7967f1) * (edit) hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java * (edit) hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/mapreduce/MapReduceBackupMergeJob.java * (edit) hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestIncrementalBackupMergeWithFailures.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java > Backup: further optimizations > - > > Key: HBASE-17825 > URL: https://issues.apache.org/jira/browse/HBASE-17825 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 3.0.0 > > Attachments: HBASE-17825-v1.patch, HBASE-17825-v2.patch, > HBASE-17825-v3.patch, HBASE-17825-v4.patch, HBASE-17825-v5.patch > > > Some phases of backup and restore can be optimized: > # WALPlayer support for multiple tables > # Run DistCp once per all tables during backup/restore > The eventual goal: > # 2 M/R jobs per backup/restore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[
https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379592#comment-16379592
]
Hadoop QA commented on HBASE-17825:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 7m
24s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
49s{color} | {color:red} hbase-mapreduce in master has 1 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
43s{color} | {color:red} hbase-backup in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} The patch hbase-server passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} hbase-mapreduce: The patch generated 0 new + 19
unchanged - 3 fixed = 19 total (was 22) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} The patch hbase-backup passed checkstyle {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
21s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 26s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 78m
13s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
57s{color} | {color:green} hbase-mapreduce in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
47s{color} | {color:green} hbase-backup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
3s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 12s{color} |
{color:black} {color}
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[ https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379587#comment-16379587 ] Ted Yu commented on HBASE-17825: I don't have other comment. > Backup: further optimizations > - > > Key: HBASE-17825 > URL: https://issues.apache.org/jira/browse/HBASE-17825 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 3.0.0 > > Attachments: HBASE-17825-v1.patch, HBASE-17825-v2.patch, > HBASE-17825-v3.patch, HBASE-17825-v4.patch, HBASE-17825-v5.patch > > > Some phases of backup and restore can be optimized: > # WALPlayer support for multiple tables > # Run DistCp once per all tables during backup/restore > The eventual goal: > # 2 M/R jobs per backup/restore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[ https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379547#comment-16379547 ] Josh Elser commented on HBASE-17825: Ya, I think I'm OK with it. Will wait for QA to come back and give Ted time to comment if he has anything else to add (since v5 was pretty much based on his review). > Backup: further optimizations > - > > Key: HBASE-17825 > URL: https://issues.apache.org/jira/browse/HBASE-17825 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 3.0.0 > > Attachments: HBASE-17825-v1.patch, HBASE-17825-v2.patch, > HBASE-17825-v3.patch, HBASE-17825-v4.patch, HBASE-17825-v5.patch > > > Some phases of backup and restore can be optimized: > # WALPlayer support for multiple tables > # Run DistCp once per all tables during backup/restore > The eventual goal: > # 2 M/R jobs per backup/restore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[ https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379537#comment-16379537 ] Vladimir Rodionov commented on HBASE-17825: --- Commit? > Backup: further optimizations > - > > Key: HBASE-17825 > URL: https://issues.apache.org/jira/browse/HBASE-17825 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 3.0.0 > > Attachments: HBASE-17825-v1.patch, HBASE-17825-v2.patch, > HBASE-17825-v3.patch, HBASE-17825-v4.patch, HBASE-17825-v5.patch > > > Some phases of backup and restore can be optimized: > # WALPlayer support for multiple tables > # Run DistCp once per all tables during backup/restore > The eventual goal: > # 2 M/R jobs per backup/restore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[
https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379382#comment-16379382
]
Josh Elser commented on HBASE-17825:
{code:java}
+List list = new ArrayList();{code}
Could set the size on the ArrayList.
Seems fine otherwise.
> Backup: further optimizations
> -
>
> Key: HBASE-17825
> URL: https://issues.apache.org/jira/browse/HBASE-17825
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
> Labels: backup
> Fix For: 3.0.0
>
> Attachments: HBASE-17825-v1.patch, HBASE-17825-v2.patch,
> HBASE-17825-v3.patch, HBASE-17825-v4.patch, HBASE-17825-v5.patch
>
>
> Some phases of backup and restore can be optimized:
> # WALPlayer support for multiple tables
> # Run DistCp once per all tables during backup/restore
> The eventual goal:
> # 2 M/R jobs per backup/restore
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[ https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379377#comment-16379377 ] Vladimir Rodionov commented on HBASE-17825: --- Patch v5 adds WALPlayer backward compatibility. > Backup: further optimizations > - > > Key: HBASE-17825 > URL: https://issues.apache.org/jira/browse/HBASE-17825 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 3.0.0 > > Attachments: HBASE-17825-v1.patch, HBASE-17825-v2.patch, > HBASE-17825-v3.patch, HBASE-17825-v4.patch, HBASE-17825-v5.patch > > > Some phases of backup and restore can be optimized: > # WALPlayer support for multiple tables > # Run DistCp once per all tables during backup/restore > The eventual goal: > # 2 M/R jobs per backup/restore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[
https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374850#comment-16374850
]
Vladimir Rodionov commented on HBASE-17825:
---
{quote}
How does this change output
{quote}
It does not change output, but it changes output directory structure. It used
to be flat, now it adds subdirectories for every table:
If we convert WALs for tables T1, T2, T3 the tool will create separate
directories for each table:
T1/
T2/
T3/
> Backup: further optimizations
> -
>
> Key: HBASE-17825
> URL: https://issues.apache.org/jira/browse/HBASE-17825
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
> Labels: backup
> Fix For: 3.0.0
>
> Attachments: HBASE-17825-v1.patch, HBASE-17825-v2.patch,
> HBASE-17825-v3.patch, HBASE-17825-v4.patch
>
>
> Some phases of backup and restore can be optimized:
> # WALPlayer support for multiple tables
> # Run DistCp once per all tables during backup/restore
> The eventual goal:
> # 2 M/R jobs per backup/restore
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[
https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373952#comment-16373952
]
Hadoop QA commented on HBASE-17825:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 7m
21s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} hbase-mapreduce: The patch generated 15 new + 21
unchanged - 2 fixed = 36 total (was 23) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} hbase-backup: The patch generated 5 new + 0 unchanged
- 0 fixed = 5 total (was 0) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
42s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
19m 42s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}105m
39s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
28s{color} | {color:green} hbase-mapreduce in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m
12s{color} | {color:green} hbase-backup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}176m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-17825 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911657/HBASE-17825-v4.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux d6e0e9522086 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[
https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373909#comment-16373909
]
stack commented on HBASE-17825:
---
bq. What are our compatibility guarantees on WALPlayer?
@InterfaceAudience.Public
public class WALPlayer extends Configured implements Tool {
A recent fix ensured hbase-1 can play hbase-2 WALs and vice-versa.
How does this change
> Backup: further optimizations
> -
>
> Key: HBASE-17825
> URL: https://issues.apache.org/jira/browse/HBASE-17825
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
> Labels: backup
> Fix For: 3.0.0
>
> Attachments: HBASE-17825-v1.patch, HBASE-17825-v2.patch,
> HBASE-17825-v3.patch, HBASE-17825-v4.patch
>
>
> Some phases of backup and restore can be optimized:
> # WALPlayer support for multiple tables
> # Run DistCp once per all tables during backup/restore
> The eventual goal:
> # 2 M/R jobs per backup/restore
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[
https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373830#comment-16373830
]
Vladimir Rodionov commented on HBASE-17825:
---
{quote}
What are our compatibility guarantees on WALPlayer?
{quote}
None. HBASE 2.0 is not backward compatible with 1.x applications to my best
knowledge.
> Backup: further optimizations
> -
>
> Key: HBASE-17825
> URL: https://issues.apache.org/jira/browse/HBASE-17825
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
> Labels: backup
> Fix For: 3.0.0
>
> Attachments: HBASE-17825-v1.patch, HBASE-17825-v2.patch,
> HBASE-17825-v3.patch, HBASE-17825-v4.patch
>
>
> Some phases of backup and restore can be optimized:
> # WALPlayer support for multiple tables
> # Run DistCp once per all tables during backup/restore
> The eventual goal:
> # 2 M/R jobs per backup/restore
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[ https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373028#comment-16373028 ] Ted Yu commented on HBASE-17825: I think we should try to provide backward compatibility. Maybe introduce a config for the new directory structure, default to the same as current structure. > Backup: further optimizations > - > > Key: HBASE-17825 > URL: https://issues.apache.org/jira/browse/HBASE-17825 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 3.0.0 > > Attachments: HBASE-17825-v1.patch, HBASE-17825-v2.patch, > HBASE-17825-v3.patch > > > Some phases of backup and restore can be optimized: > # WALPlayer support for multiple tables > # Run DistCp once per all tables during backup/restore > The eventual goal: > # 2 M/R jobs per backup/restore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[ https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372915#comment-16372915 ] Josh Elser commented on HBASE-17825: What are our compatibility guarantees on WALPlayer? The test failure makes me realize that if users are relying on output in a certain way, this change will break them. Not sure if this is something to be concerned about. I assume you'll be looking at the other things QA called out, Vlad. > Backup: further optimizations > - > > Key: HBASE-17825 > URL: https://issues.apache.org/jira/browse/HBASE-17825 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 3.0.0 > > Attachments: HBASE-17825-v1.patch, HBASE-17825-v2.patch, > HBASE-17825-v3.patch > > > Some phases of backup and restore can be optimized: > # WALPlayer support for multiple tables > # Run DistCp once per all tables during backup/restore > The eventual goal: > # 2 M/R jobs per backup/restore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[
https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372384#comment-16372384
]
Hadoop QA commented on HBASE-17825:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
23s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} hbase-mapreduce: The patch generated 15 new + 20
unchanged - 2 fixed = 35 total (was 22) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} hbase-backup: The patch generated 5 new + 0 unchanged
- 0 fixed = 5 total (was 0) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
6s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m
1s{color} | {color:red} The patch causes 10 errors with Hadoop v2.6.5. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m
7s{color} | {color:red} The patch causes 10 errors with Hadoop v2.7.4. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m
28s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 5s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 11m 44s{color}
| {color:red} hbase-mapreduce in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m
9s{color} | {color:green} hbase-backup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}159m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestRegionOpen |
| | hadoop.hbase.mapreduce.TestWALPlayer |
\\
\\
|| Subsystem ||
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[ https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361216#comment-16361216 ] Josh Elser commented on HBASE-17825: Looks fine to me. Let's see what Precommit says. > Backup: further optimizations > - > > Key: HBASE-17825 > URL: https://issues.apache.org/jira/browse/HBASE-17825 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 3.0.0 > > Attachments: HBASE-17825-v1.patch, HBASE-17825-v2.patch > > > Some phases of backup and restore can be optimized: > # WALPlayer support for multiple tables > # Run DistCp once per all tables during backup/restore > The eventual goal: > # 2 M/R jobs per backup/restore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[
https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361202#comment-16361202
]
Hadoop QA commented on HBASE-17825:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 3s{color}
| {color:red} HBASE-17825 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.7.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-17825 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12909993/HBASE-17825-v2.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/11496/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Backup: further optimizations
> -
>
> Key: HBASE-17825
> URL: https://issues.apache.org/jira/browse/HBASE-17825
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
> Labels: backup
> Fix For: 3.0.0
>
> Attachments: HBASE-17825-v1.patch, HBASE-17825-v2.patch
>
>
> Some phases of backup and restore can be optimized:
> # WALPlayer support for multiple tables
> # Run DistCp once per all tables during backup/restore
> The eventual goal:
> # 2 M/R jobs per backup/restore
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[ https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359076#comment-16359076 ] Vladimir Rodionov commented on HBASE-17825: --- Patch v2 contains minor fixes. Created HBASE-19969 to improve FT during Merge > Backup: further optimizations > - > > Key: HBASE-17825 > URL: https://issues.apache.org/jira/browse/HBASE-17825 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 3.0.0 > > Attachments: HBASE-17825-v1.patch, HBASE-17825-v2.patch > > > Some phases of backup and restore can be optimized: > # WALPlayer support for multiple tables > # Run DistCp once per all tables during backup/restore > The eventual goal: > # 2 M/R jobs per backup/restore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[
https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359031#comment-16359031
]
Vladimir Rodionov commented on HBASE-17825:
---
{quote}
When false is returned, some files would be deleted while some are left. Is
this desirable?
{quote}
This needs to be added to FT Jira. We need to restore not only meta in case of
failure but physical data as well.
> Backup: further optimizations
> -
>
> Key: HBASE-17825
> URL: https://issues.apache.org/jira/browse/HBASE-17825
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
> Labels: backup
> Fix For: 3.0.0
>
> Attachments: HBASE-17825-v1.patch
>
>
> Some phases of backup and restore can be optimized:
> # WALPlayer support for multiple tables
> # Run DistCp once per all tables during backup/restore
> The eventual goal:
> # 2 M/R jobs per backup/restore
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[ https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357641#comment-16357641 ] Josh Elser commented on HBASE-17825: bq. I think it is fine, this is work in progress and it has not been released yet to public. Yeah, not a big deal at this point. We can include a warning as a release note, but not a big problem if this is a necessary change as you've outlined. > Backup: further optimizations > - > > Key: HBASE-17825 > URL: https://issues.apache.org/jira/browse/HBASE-17825 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 3.0.0 > > Attachments: HBASE-17825-v1.patch > > > Some phases of backup and restore can be optimized: > # WALPlayer support for multiple tables > # Run DistCp once per all tables during backup/restore > The eventual goal: > # 2 M/R jobs per backup/restore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[
https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357637#comment-16357637
]
Vladimir Rodionov commented on HBASE-17825:
---
{quote}
This is just "in-flight" backups, so there's no concern with any existing
backup data, right?
{quote}
No, this changes directory structure in backup FS. I think it is fine, this is
work in progress and it has not been released yet to public.
> Backup: further optimizations
> -
>
> Key: HBASE-17825
> URL: https://issues.apache.org/jira/browse/HBASE-17825
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
> Labels: backup
> Fix For: 3.0.0
>
> Attachments: HBASE-17825-v1.patch
>
>
> Some phases of backup and restore can be optimized:
> # WALPlayer support for multiple tables
> # Run DistCp once per all tables during backup/restore
> The eventual goal:
> # 2 M/R jobs per backup/restore
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[ https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357561#comment-16357561 ] Josh Elser commented on HBASE-17825: bq. I do not remember exactly, but it was necessary to remove this data subdirectory to make the patch work. Anyway, I simplified the structure of backup file system. Is this documented somewhere that should be updated to reflect the new structure? It doesn't ring any bells to me. This is just "in-flight" backups, so there's no concern with any existing backup data, right? > Backup: further optimizations > - > > Key: HBASE-17825 > URL: https://issues.apache.org/jira/browse/HBASE-17825 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 3.0.0 > > Attachments: HBASE-17825-v1.patch > > > Some phases of backup and restore can be optimized: > # WALPlayer support for multiple tables > # Run DistCp once per all tables during backup/restore > The eventual goal: > # 2 M/R jobs per backup/restore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[
https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357516#comment-16357516
]
Vladimir Rodionov commented on HBASE-17825:
---
{quote}
Why change directory ?
{quote}
I do not remember exactly, but it was necessary to remove this *data*
subdirectory to make the patch work. Anyway, I simplified the structure of
backup file system.
> Backup: further optimizations
> -
>
> Key: HBASE-17825
> URL: https://issues.apache.org/jira/browse/HBASE-17825
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
> Labels: backup
> Fix For: 3.0.0
>
> Attachments: HBASE-17825-v1.patch
>
>
> Some phases of backup and restore can be optimized:
> # WALPlayer support for multiple tables
> # Run DistCp once per all tables during backup/restore
> The eventual goal:
> # 2 M/R jobs per backup/restore
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[
https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357357#comment-16357357
]
Josh Elser commented on HBASE-17825:
{code}
+ if (fileName.indexOf(".tabledesc") > 0 ||
+ fileName.indexOf(".regioninfo") > 0 ) {
{code}
Do we have constants elsewhere we can use instead of the literal strings?
{code}
TableMapReduceUtil.addDependencyJarsForClasses(job.getConfiguration(),
-
org.apache.hbase.thirdparty.com.google.common.base.Preconditions.class);
+
org.apache.hbase.thirdparty.com.google.common.base.Preconditions.class);
{code}
{{TableMapReduceUtil.addHBaseDependencyJars}} should be doing this already for
you. I think you should be able to drop this right now, but maybe it's good to
preserve to prevent any breakage in the future? Not sure..
Also curious about the change in data directory that Ted pointed out.
> Backup: further optimizations
> -
>
> Key: HBASE-17825
> URL: https://issues.apache.org/jira/browse/HBASE-17825
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
> Labels: backup
> Fix For: 3.0.0
>
> Attachments: HBASE-17825-v1.patch
>
>
> Some phases of backup and restore can be optimized:
> # WALPlayer support for multiple tables
> # Run DistCp once per all tables during backup/restore
> The eventual goal:
> # 2 M/R jobs per backup/restore
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[
https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356372#comment-16356372
]
Ted Yu commented on HBASE-17825:
{code}
-new Path(HBackupFileSystem.getTableBackupDataDir(backupRoot,
mergedBackupId, tableName));
+new Path(HBackupFileSystem.getTableBackupDir(backupRoot,
mergedBackupId, tableName));
{code}
Why change directory ?
{code}
+ boolean result = fs.delete(p, false);
+ if (!result) {
+return false;
{code}
When false is returned, some files would be deleted while some are left. Is
this desirable ?
{code}
+String[] tableNameParts = tableName.split(":");
+Path tableRelPath = new Path(tableName.split(":")[0]);
{code}
You can use tableNameParts instead of calling split() multiple times.
{code}
+byte[] outKey =
+Bytes.add(table.getName(), Bytes.toBytes(tableSeparator),
CellUtil.cloneRow(cell));
{code}
Is the above a compatible change ?
{code}
+ List tableNames = getTableNameList(tables);
{code}
After you obtain the List, you iterate thru tableNames. Looks like creation of
the List can be avoided.
> Backup: further optimizations
> -
>
> Key: HBASE-17825
> URL: https://issues.apache.org/jira/browse/HBASE-17825
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
> Labels: backup
> Fix For: 3.0.0
>
> Attachments: HBASE-17825-v1.patch
>
>
> Some phases of backup and restore can be optimized:
> # WALPlayer support for multiple tables
> # Run DistCp once per all tables during backup/restore
> The eventual goal:
> # 2 M/R jobs per backup/restore
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[ https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356347#comment-16356347 ] Vladimir Rodionov commented on HBASE-17825: --- Patch v1 improves incremental backup run time. Now we have only 2 (two) MR jobs instead of N+1, where N is the number of tables in a backup operation. cc: [~elserj], [~te...@apache.org] > Backup: further optimizations > - > > Key: HBASE-17825 > URL: https://issues.apache.org/jira/browse/HBASE-17825 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 3.0.0 > > Attachments: HBASE-17825-v1.patch > > > Some phases of backup and restore can be optimized: > # WALPlayer support for multiple tables > # Run DistCp once per all tables during backup/restore > The eventual goal: > # 2 M/R jobs per backup/restore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[ https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252078#comment-16252078 ] Josh Elser commented on HBASE-17825: Hrm, sorry [~vrodionov], maybe I'm crossing wires? I thought HBASE-15227 captured the remaining implementation changes you were working on. Didn't realize that this one was also in that bucket. > Backup: further optimizations > - > > Key: HBASE-17825 > URL: https://issues.apache.org/jira/browse/HBASE-17825 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 2.1.0 > > > Some phases of backup and restore can be optimized: > # WALPlayer support for multiple tables > # Run DistCp once per all tables during backup/restore > The eventual goal: > # 2 M/R jobs per backup/restore -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[ https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252073#comment-16252073 ] Vladimir Rodionov commented on HBASE-17825: --- Why, it is almost done, [~elserj]. Phase 4 is post 2.0 anyway. > Backup: further optimizations > - > > Key: HBASE-17825 > URL: https://issues.apache.org/jira/browse/HBASE-17825 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 2.1.0 > > > Some phases of backup and restore can be optimized: > # WALPlayer support for multiple tables > # Run DistCp once per all tables during backup/restore > The eventual goal: > # 2 M/R jobs per backup/restore -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[ https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252062#comment-16252062 ] Josh Elser commented on HBASE-17825: Hrm, IMO, let's pessimistically kick this out to Phase 4. If it happens that we do have time to do this, we can bring it back. > Backup: further optimizations > - > > Key: HBASE-17825 > URL: https://issues.apache.org/jira/browse/HBASE-17825 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > > Some phases of backup and restore can be optimized: > # WALPlayer support for multiple tables > # Run DistCp once per all tables during backup/restore > The eventual goal: > # 2 M/R jobs per backup/restore -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[ https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248178#comment-16248178 ] Vladimir Rodionov commented on HBASE-17825: --- Yes, I think. I will return to this when we finish testing. > Backup: further optimizations > - > > Key: HBASE-17825 > URL: https://issues.apache.org/jira/browse/HBASE-17825 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > > Some phases of backup and restore can be optimized: > # WALPlayer support for multiple tables > # Run DistCp once per all tables during backup/restore > The eventual goal: > # 2 M/R jobs per backup/restore -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[ https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248115#comment-16248115 ] Josh Elser commented on HBASE-17825: Is the "In progress" still accurate, [~vrodionov]? (something you're shooting for 2.0.0-beta-1) > Backup: further optimizations > - > > Key: HBASE-17825 > URL: https://issues.apache.org/jira/browse/HBASE-17825 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > > Some phases of backup and restore can be optimized: > # WALPlayer support for multiple tables > # Run DistCp once per all tables during backup/restore > The eventual goal: > # 2 M/R jobs per backup/restore -- This message was sent by Atlassian JIRA (v6.4.14#64029)
