[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204425#comment-16204425 ] Hudson commented on HBASE-18843: FAILURE: Integrated in Jenkins build HBase-2.0 #682 (See [https://builds.apache.org/job/HBase-2.0/682/]) HBASE-18843 Add DistCp support to incremental backup with bulk loading (tedyu: rev a4bc3c6db7bad8832c953d809e5a09c94297860f) * (edit) hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestBackupBase.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/mapreduce/MapReduceBackupCopyJob.java > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch, > HBASE-18843-v4.patch, HBASE-18843-v5.patch, HBASE-18843-v6.patch, > HBASE-18843-v7.patch, HBASE-18843-v8.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[
https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204249#comment-16204249
]
Hadoop QA commented on HBASE-18843:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
1s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
54s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
46m 52s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
47s{color} | {color:green} hbase-backup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:5d60123 |
| JIRA Issue | HBASE-18843 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12892131/HBASE-18843-v8.patch |
| Optional Tests | asflicense shadedjars javac javadoc unit findbugs
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 861daeeace94 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / ffdc0e6 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/9110/testReport/ |
| modules | C: hbase-backup U: hbase-backup |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/9110/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> Add DistCp support to incremental backup with bulk loading
>
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204091#comment-16204091 ] Vladimir Rodionov commented on HBASE-18843: --- Thanks, [~elserj]. I opened HBASE-19906 to address hadoop3 issue. > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch, > HBASE-18843-v4.patch, HBASE-18843-v5.patch, HBASE-18843-v6.patch, > HBASE-18843-v7.patch, HBASE-18843-v8.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204063#comment-16204063 ] Josh Elser commented on HBASE-18843: Thanks, Vlad! I had a chat with Vlad earlier today on this issue. Vlad's take was that Distcp is busted in Hadoop3 presently -- stacktraces down in the weeds of Hadoop that we haven't yet unraveled. I don't want us to be in a situation where this is "held hostage" by Hadoop3 bugs that need to be worked out, but I also want to make sure that we don't ignore Hadoop3 issues when we know about 'em. My opinion is that we should get a follow-on issues filed (I guess here in HBASE for now, can switch it to the appropriate Hadoop JIRA project after investigation) for the Hadoop3 runtime issue, and get this committed. > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch, > HBASE-18843-v4.patch, HBASE-18843-v5.patch, HBASE-18843-v6.patch, > HBASE-18843-v7.patch, HBASE-18843-v8.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[
https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195273#comment-16195273
]
Vladimir Rodionov commented on HBASE-18843:
---
{quote}
Fix from HDFS-12599 is needed.
You can omit patch from HADOOP-14930 by specifying
-Djetty.version=9.3.19.v20170502 on the command line.
{quote}
[~te...@apache.org], do we have a separate JIRA for this issue?
> Add DistCp support to incremental backup with bulk loading
> --
>
> Key: HBASE-18843
> URL: https://issues.apache.org/jira/browse/HBASE-18843
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch,
> HBASE-18843-v4.patch, HBASE-18843-v5.patch, HBASE-18843-v6.patch,
> HBASE-18843-v7.patch
>
>
> Currently, we copy bulk loaded files to backup one-by-one on a client side
> (where backup create runs). This has to be replaced with DistCp copying.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[
https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195093#comment-16195093
]
Hudson commented on HBASE-18843:
Results for branch HBASE-18467, done in 4 hr 24 min and counting
[build #136 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18467/136/]:
FAILURE
details (if available):
(x) *{color:red}-1 overall{color}*
Committer, please check your recent inclusion of a patch for this issue.
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18467/136//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 checks{color}
-- For more information [see jdk8
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18467/136//JDK8_Nightly_Build_Report/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Add DistCp support to incremental backup with bulk loading
> --
>
> Key: HBASE-18843
> URL: https://issues.apache.org/jira/browse/HBASE-18843
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch,
> HBASE-18843-v4.patch, HBASE-18843-v5.patch, HBASE-18843-v6.patch,
> HBASE-18843-v7.patch
>
>
> Currently, we copy bulk loaded files to backup one-by-one on a client side
> (where backup create runs). This has to be replaced with DistCp copying.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194771#comment-16194771 ] Ted Yu commented on HBASE-18843: Fix from HDFS-12599 is needed. You can omit patch from HADOOP-14930 by specifying -Djetty.version=9.3.19.v20170502 on the command line. > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch, > HBASE-18843-v4.patch, HBASE-18843-v5.patch, HBASE-18843-v6.patch, > HBASE-18843-v7.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[
https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194041#comment-16194041
]
Vladimir Rodionov commented on HBASE-18843:
---
{quote}
Apply patches from HDFS-12599 and HADOOP-14930
{quote}
Who is going to take care of this issue on a Hadoop side? It seems that w/o
these patches HBase 2.0 is blocked on Hadoop3?
> Add DistCp support to incremental backup with bulk loading
> --
>
> Key: HBASE-18843
> URL: https://issues.apache.org/jira/browse/HBASE-18843
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch,
> HBASE-18843-v4.patch, HBASE-18843-v5.patch, HBASE-18843-v6.patch,
> HBASE-18843-v7.patch
>
>
> Currently, we copy bulk loaded files to backup one-by-one on a client side
> (where backup create runs). This has to be replaced with DistCp copying.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194034#comment-16194034 ] Ted Yu commented on HBASE-18843: Checkout branch-3.0.0-beta1 branch of hadoop Apply patches from HDFS-12599 and HADOOP-14930 'mvn install -DskipTests' Then go to hbase master branch, use the two mvn commands shown above. > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch, > HBASE-18843-v4.patch, HBASE-18843-v5.patch, HBASE-18843-v6.patch, > HBASE-18843-v7.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194030#comment-16194030 ] Vladimir Rodionov commented on HBASE-18843: --- [~tedyu], how can I reproduce this on master? > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch, > HBASE-18843-v4.patch, HBASE-18843-v5.patch, HBASE-18843-v6.patch, > HBASE-18843-v7.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[
https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192192#comment-16192192
]
Ted Yu commented on HBASE-18843:
Even though the above error can be reproduced on command line, every time I
debug in Eclipse, hadoop 2.7.1 is used:
{code}
2017-10-04 16:16:36,709 INFO [main] log.Slf4jLog(67): Extract
jar:file:/Users/tyu/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.1/hadoop-hdfs-2.7.1-tests.jar!/webapps/datanode
to
/var/folders/4g/2vdss5497xx9blpn2pbqc38rgn/T/Jetty_localhost_55722_datanode.nq742/webapp
{code}
Still hunting for the cause.
> Add DistCp support to incremental backup with bulk loading
> --
>
> Key: HBASE-18843
> URL: https://issues.apache.org/jira/browse/HBASE-18843
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch,
> HBASE-18843-v4.patch, HBASE-18843-v5.patch, HBASE-18843-v6.patch,
> HBASE-18843-v7.patch
>
>
> Currently, we copy bulk loaded files to backup one-by-one on a client side
> (where backup create runs). This has to be replaced with DistCp copying.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192039#comment-16192039 ] Ted Yu commented on HBASE-18843: I ran the test using command given by Vlad but with the following (without patch): -Dhadoop.version=3.0.0-beta1 TestIncrementalBackup passed. 3.0.0-beta1 maven artifact was populated yesterday. > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch, > HBASE-18843-v4.patch, HBASE-18843-v5.patch, HBASE-18843-v6.patch, > HBASE-18843-v7.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[
https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16191839#comment-16191839
]
Hadoop QA commented on HBASE-18843:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
46s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
30s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
11s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
39m 35s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
35s{color} | {color:green} hbase-backup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:5d60123 |
| JIRA Issue | HBASE-18843 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12890401/HBASE-18843-v7.patch |
| Optional Tests | asflicense shadedjars javac javadoc unit findbugs
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 2ea33a50ce58 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 11aa674 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8941/testReport/ |
| modules | C: hbase-backup U: hbase-backup |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/8941/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> Add DistCp support to incremental backup with bulk loading
>
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[
https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16191831#comment-16191831
]
Vladimir Rodionov commented on HBASE-18843:
---
What I did:
git pull
mvn clean install -DskipTests -Dhadoop.version=3.0
mvn test -Dtest=org.apache.hadoop.hbase.backup.TestIncrementalBackup
-Dhadoop.profile=3.0
{code}
017-10-04 11:47:57,232 ERROR [main] zookeeper.RecoverableZooKeeper(300):
ZooKeeper exists failed after 4 attempts
2017-10-04 11:47:57,233 WARN [main] zookeeper.ZKUtil(416):
hconnection-0xa77614d0x0, quorum=localhost:21818, baseZNode=/1 Unable to set
watcher on znode (/1/hbaseid)
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for /1/hbaseid
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:231)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:413)
at
org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:102)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:288)
{code}
The same issue I see when run other backup tests. I do not think, this
backup-related and it should not prevent from committing this patch.
> Add DistCp support to incremental backup with bulk loading
> --
>
> Key: HBASE-18843
> URL: https://issues.apache.org/jira/browse/HBASE-18843
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch,
> HBASE-18843-v4.patch, HBASE-18843-v5.patch, HBASE-18843-v6.patch,
> HBASE-18843-v7.patch
>
>
> Currently, we copy bulk loaded files to backup one-by-one on a client side
> (where backup create runs). This has to be replaced with DistCp copying.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[
https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16191691#comment-16191691
]
Josh Elser commented on HBASE-18843:
Sorry! Forgot to reply.
On master (0a24178d06ac3dd5780522d52b5054e7aff1e50f), I tried to run tests with
{{-Dhadoop.profile=3.0}}. I see some deterministic failures (some around Jetty
and Kerby method signature changes), but I haven't seen all minicluster test
fail.
Do you have a specific commit + command you're running which I can try?
> Add DistCp support to incremental backup with bulk loading
> --
>
> Key: HBASE-18843
> URL: https://issues.apache.org/jira/browse/HBASE-18843
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch,
> HBASE-18843-v4.patch, HBASE-18843-v5.patch, HBASE-18843-v6.patch
>
>
> Currently, we copy bulk loaded files to backup one-by-one on a client side
> (where backup create runs). This has to be replaced with DistCp copying.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16191638#comment-16191638 ] Vladimir Rodionov commented on HBASE-18843: --- [~elserj], were you able to reproduce the issue? > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch, > HBASE-18843-v4.patch, HBASE-18843-v5.patch, HBASE-18843-v6.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16191344#comment-16191344 ] Josh Elser commented on HBASE-18843: bq. HBase minicluster fails to start - all backup tests fail. Not sure, how to run it under 3.0. Hrm, maybe master is just busted (something landed on branch-2 but not master?). Let me see if I can repro. > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch, > HBASE-18843-v4.patch, HBASE-18843-v5.patch, HBASE-18843-v6.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[
https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190747#comment-16190747
]
Vladimir Rodionov commented on HBASE-18843:
---
{quote}
You file something for this yet (I didn't see anything)?
{quote}
HBase minicluster fails to start - all backup tests fail. Not sure, how to run
it under 3.0. I built hbase locally with hadoop3.0 profile and then tried to
run tests. Some Zk issues resulted in mini cluster shutdown.
> Add DistCp support to incremental backup with bulk loading
> --
>
> Key: HBASE-18843
> URL: https://issues.apache.org/jira/browse/HBASE-18843
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch,
> HBASE-18843-v4.patch, HBASE-18843-v5.patch, HBASE-18843-v6.patch
>
>
> Currently, we copy bulk loaded files to backup one-by-one on a client side
> (where backup create runs). This has to be replaced with DistCp copying.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[
https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190746#comment-16190746
]
Hadoop QA commented on HBASE-18843:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m
43s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
24s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
58s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
36m 50s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
38s{color} | {color:red} hbase-backup generated 3 new + 0 unchanged - 0 fixed =
3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
43s{color} | {color:green} hbase-backup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 70m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-backup |
| | Potentially ambiguous invocation of either an outer or inherited method
org.apache.hadoop.conf.Configured.getConf() in
org.apache.hadoop.hbase.backup.mapreduce.MapReduceBackupCopyJob$BackupDistCp.createInputFileListing(Job)
At MapReduceBackupCopyJob.java:an outer or inherited method
org.apache.hadoop.conf.Configured.getConf() in
org.apache.hadoop.hbase.backup.mapreduce.MapReduceBackupCopyJob$BackupDistCp.createInputFileListing(Job)
At MapReduceBackupCopyJob.java:[line 287] |
| | Potentially ambiguous invocation of either an outer or inherited method
org.apache.hadoop.conf.Configured.getConf() in
org.apache.hadoop.hbase.backup.mapreduce.MapReduceBackupCopyJob$BackupDistCp.getKey(Path)
At MapReduceBackupCopyJob.java:an outer or inherited method
org.apache.hadoop.conf.Configured.getConf() in
org.apache.hadoop.hbase.backup.mapreduce.MapReduceBackupCopyJob$BackupDistCp.getKey(Path)
At MapReduceBackupCopyJob.java:[line 326] |
| | Potentially ambiguous invocation of either an outer or inherited method
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190730#comment-16190730 ] Josh Elser commented on HBASE-18843: bq. all backup tests fail under 3.0 - totally separate issue You file something for this yet (I didn't see anything)? Think we should get the tests working on 3.0 and then come back to this issue? > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch, > HBASE-18843-v4.patch, HBASE-18843-v5.patch, HBASE-18843-v6.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[
https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190701#comment-16190701
]
Ted Yu commented on HBASE-18843:
{code}
+LOG.error(t);
+t.printStackTrace();
{code}
printStackTrace() should be dropped.
{code}
+writer.close();
{code}
Put the close() call in finally block.
> Add DistCp support to incremental backup with bulk loading
> --
>
> Key: HBASE-18843
> URL: https://issues.apache.org/jira/browse/HBASE-18843
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch,
> HBASE-18843-v4.patch, HBASE-18843-v5.patch, HBASE-18843-v6.patch
>
>
> Currently, we copy bulk loaded files to backup one-by-one on a client side
> (where backup create runs). This has to be replaced with DistCp copying.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184564#comment-16184564 ] Josh Elser commented on HBASE-18843: Filed HBASE-18903 to figure out what went wrong with the PreCommit. > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch, > HBASE-18843-v4.patch, HBASE-18843-v5.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183850#comment-16183850 ] Hudson commented on HBASE-18843: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3791 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3791/]) Revert HBASE-18843. Needs more work. Also, breaks building with Hadoop3. (appy: rev 2bf5bbd0a2a4a787b6e83615798d43f1526d8956) * (delete) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/FixedRelativePathCopyListing.java * (edit) hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestBackupBase.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/mapreduce/MapReduceBackupCopyJob.java > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch, > HBASE-18843-v4.patch, HBASE-18843-v5.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183820#comment-16183820 ] Hudson commented on HBASE-18843: FAILURE: Integrated in Jenkins build HBase-2.0 #590 (See [https://builds.apache.org/job/HBase-2.0/590/]) Revert HBASE-18843. Needs more work. Also, breaks building with Hadoop3. (appy: rev 75febcea89c907a7daf70c0a06b92803accc3799) * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java * (delete) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/FixedRelativePathCopyListing.java * (edit) hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestBackupBase.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/mapreduce/MapReduceBackupCopyJob.java > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch, > HBASE-18843-v4.patch, HBASE-18843-v5.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183636#comment-16183636 ] Josh Elser commented on HBASE-18843: bq. Doing this now. bq. Reverted from branch-2 and master. Oh... I guess I won't revert it then. I had builds running locally to make sure the branches were actually "good". > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch, > HBASE-18843-v4.patch, HBASE-18843-v5.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183634#comment-16183634 ] Appy commented on HBASE-18843: -- Reverted from branch-2 and master. > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch, > HBASE-18843-v4.patch, HBASE-18843-v5.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[
https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183624#comment-16183624
]
Appy commented on HBASE-18843:
--
Thanks a lot [~elserj] for prompt back-and-forth! Yeah, let's revert it for now.
I looked around a bit more, and maybe have a more elegant solution.
Just override {{protected void doBuildListing(Path pathToListingFile,
DistCpContext context) throws IOException }} which is marked protected, so is
meant to be.
For each path, do path=path.getParent() for 1 less time than is being done in
current {{computeSourceRootPath}}. Set the context appropriately so that the
default implementation of {{computeSourceRootPath}} does one more
path.getParent. And that should be enough!
> Add DistCp support to incremental backup with bulk loading
> --
>
> Key: HBASE-18843
> URL: https://issues.apache.org/jira/browse/HBASE-18843
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0-alpha-4
>
> Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch,
> HBASE-18843-v4.patch, HBASE-18843-v5.patch
>
>
> Currently, we copy bulk loaded files to backup one-by-one on a client side
> (where backup create runs). This has to be replaced with DistCp copying.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183617#comment-16183617 ] Josh Elser commented on HBASE-18843: bq. I guess followup is fine. Nah, you're right. It was an unintended de-stabilization. Let's revert for now. This isn't blocking folks if it's *not* in the tree. Doing this now. > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch, > HBASE-18843-v4.patch, HBASE-18843-v5.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183614#comment-16183614 ] Appy commented on HBASE-18843: -- My main concern is, if Vlad is not around in some years, there's no knowing why he did what he did. And it'll be a mess for anyone to figure it out for any refactoring/deletion/etc. > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch, > HBASE-18843-v4.patch, HBASE-18843-v5.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[
https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183613#comment-16183613
]
Appy commented on HBASE-18843:
--
I guess followup is fine.
But i have a bigger question, why is this class such almost duplicate of
SimpleCopyListing. I see that [~tedyu] also raised that question and the answer
was - need to overwrite {{computeSourceRootPath}}. But why?
And if so, all you needed was to just override the only method which actually
calls it - {{public void doBuildListing(SequenceFile.Writer fileListWriter,
DistCpContext options) throws IOException}}
I spent sometime looking around. I see the class is dynamically loaded by
setting a conf. When does it come into picture?
Can we do the custom logic before triggering copy, and inject it somehow?
In any case, it's not good that such a blatant copy of code got checked in and
the class doesn't have a big fat comment explaining 'Why the need? What's was
the exact pain point?, etc etc'.
> Add DistCp support to incremental backup with bulk loading
> --
>
> Key: HBASE-18843
> URL: https://issues.apache.org/jira/browse/HBASE-18843
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0-alpha-4
>
> Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch,
> HBASE-18843-v4.patch, HBASE-18843-v5.patch
>
>
> Currently, we copy bulk loaded files to backup one-by-one on a client side
> (where backup create runs). This has to be replaced with DistCp copying.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183582#comment-16183582 ] Josh Elser commented on HBASE-18843: bq. The one which was supposed to test building with hadoop3 Hah! I thought it was curious that we didn't catch this :) [~appy] you think we should revert temporarily to get this fixed instead of handling as a follow-on fix? (was my original thought -- see the linked issue) > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch, > HBASE-18843-v4.patch, HBASE-18843-v5.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183576#comment-16183576 ] Appy commented on HBASE-18843: -- Not your fault [~vrodionov], our precommit is wrong. The one which was supposed to test building with hadoop3 (https://builds.apache.org/job/PreCommit-HBASE-Build/8813/artifact/patchprocess/patch-javac-3.0.0-alpha4.txt/*view*/) was still downloading 2.7.1 jars. > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch, > HBASE-18843-v4.patch, HBASE-18843-v5.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[
https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183574#comment-16183574
]
Appy commented on HBASE-18843:
--
Fails to build with hadoop3.
{noformat}
$ mvn clean install -DskipTests -Dhadoop.profile=3.0
...
...
-
[ERROR] COMPILATION ERROR :
-
[ERROR]
/Users/appy/apache/hbase/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/FixedRelativePathCopyListing.java:[76,3]
method does not override or implement a method from a supertype
[ERROR]
/Users/appy/apache/hbase/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/FixedRelativePathCopyListing.java:[98,3]
method does not override or implement a method from a supertype
[ERROR]
/Users/appy/apache/hbase/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/FixedRelativePathCopyListing.java:[107,50]
cannot find symbol
symbol: method shouldPreserveRawXattrs()
location: variable options of type org.apache.hadoop.tools.DistCpOptions
[ERROR]
/Users/appy/apache/hbase/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/FixedRelativePathCopyListing.java:[117,24]
method toCopyListingFileStatus in class
org.apache.hadoop.tools.util.DistCpUtils cannot be applied to given types;
required:
org.apache.hadoop.fs.FileSystem,org.apache.hadoop.fs.FileStatus,boolean,boolean,boolean,int
found:
org.apache.hadoop.fs.FileSystem,org.apache.hadoop.fs.FileStatus,boolean,boolean,boolean
reason: actual and formal argument lists differ in length
[ERROR]
/Users/appy/apache/hbase/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/FixedRelativePathCopyListing.java:[128,26]
method toCopyListingFileStatus in class
org.apache.hadoop.tools.util.DistCpUtils cannot be applied to given types;
required:
org.apache.hadoop.fs.FileSystem,org.apache.hadoop.fs.FileStatus,boolean,boolean,boolean,int
found:
org.apache.hadoop.fs.FileSystem,org.apache.hadoop.fs.FileStatus,boolean,boolean,boolean
reason: actual and formal argument lists differ in length
[ERROR]
/Users/appy/apache/hbase/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/FixedRelativePathCopyListing.java:[170,3]
method does not override or implement a method from a supertype
[ERROR]
/Users/appy/apache/hbase/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/FixedRelativePathCopyListing.java:[222,46]
cannot find symbol
symbol: method shouldPreserveRawXattrs()
location: variable options of type org.apache.hadoop.tools.DistCpOptions
[ERROR]
/Users/appy/apache/hbase/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/FixedRelativePathCopyListing.java:[232,22]
method toCopyListingFileStatus in class
org.apache.hadoop.tools.util.DistCpUtils cannot be applied to given types;
required:
org.apache.hadoop.fs.FileSystem,org.apache.hadoop.fs.FileStatus,boolean,boolean,boolean,int
found:
org.apache.hadoop.fs.FileSystem,org.apache.hadoop.fs.FileStatus,boolean,boolean,boolean
reason: actual and formal argument lists differ in length
[ERROR]
/Users/appy/apache/hbase/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/FixedRelativePathCopyListing.java:[273,25]
incompatible types: org.apache.hadoop.tools.CopyListingFileStatus cannot be
converted to org.apache.hadoop.fs.FileStatus
9 errors
{noformat}
> Add DistCp support to incremental backup with bulk loading
> --
>
> Key: HBASE-18843
> URL: https://issues.apache.org/jira/browse/HBASE-18843
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0-alpha-4
>
> Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch,
> HBASE-18843-v4.patch, HBASE-18843-v5.patch
>
>
> Currently, we copy bulk loaded files to backup one-by-one on a client side
> (where backup create runs). This has to be replaced with DistCp copying.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183568#comment-16183568 ] Josh Elser commented on HBASE-18843: Looks like Hadoop made a non-backwards compatible change affecting DistCpOptions in 3.0.0-alpha4 in HADOOP-14267. It appears that we'll have to work around this one with reflection (goody..) > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch, > HBASE-18843-v4.patch, HBASE-18843-v5.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182709#comment-16182709 ] Hudson commented on HBASE-18843: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3787 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3787/]) HBASE-18843 Add DistCp support to incremental backup with bulk loading (tedyu: rev 5df2fe706338c9fdee397284c51df34ddae4cfe7) * (add) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/FixedRelativePathCopyListing.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/mapreduce/MapReduceBackupCopyJob.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java * (edit) hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestBackupBase.java HBASE-18843 Add DistCp support to incremental backup with bulk loading - (tedyu: rev 976ef508f7c6f758761e438ade59d222f5cb0aa8) * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/mapreduce/MapReduceBackupCopyJob.java * (delete) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/FixedRelativePathCopyListing.java * (edit) hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestBackupBase.java HBASE-18843 Add DistCp support to incremental backup with bulk loading (tedyu: rev 417dad646bf91922fe0117c83ea69aa501f3c2fc) * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/mapreduce/MapReduceBackupCopyJob.java * (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java * (add) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/FixedRelativePathCopyListing.java * (edit) hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestBackupBase.java > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch, > HBASE-18843-v4.patch, HBASE-18843-v5.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[
https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181970#comment-16181970
]
Hadoop QA commented on HBASE-18843:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
52s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
22s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
32m 30s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
2s{color} | {color:green} hbase-backup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:5d60123 |
| JIRA Issue | HBASE-18843 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12889215/HBASE-18843-v5.patch |
| Optional Tests | asflicense shadedjars javac javadoc unit findbugs
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 6c9d7baf3b7e 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 845b83b |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8813/testReport/ |
| modules | C: hbase-backup U: hbase-backup |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/8813/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> Add DistCp support to incremental backup with bulk loading
>
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181940#comment-16181940 ] Ted Yu commented on HBASE-18843: +1 > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch, > HBASE-18843-v4.patch, HBASE-18843-v5.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[
https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181915#comment-16181915
]
Hadoop QA commented on HBASE-18843:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
25s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
21s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
39m 40s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
50s{color} | {color:green} hbase-backup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:5d60123 |
| JIRA Issue | HBASE-18843 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12889195/HBASE-18843-v4.patch |
| Optional Tests | asflicense shadedjars javac javadoc unit findbugs
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 9b028d3901e6 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 91e1f83 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8810/testReport/ |
| modules | C: hbase-backup U: hbase-backup |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/8810/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> Add DistCp support to incremental backup with bulk loading
>
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177102#comment-16177102 ] Vladimir Rodionov commented on HBASE-18843: --- Yetus-Petus, Imitation of active community involvement? Now, I have to spend my time trying to fix eclipse support, which is totally broken in the master. > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176826#comment-16176826 ] Josh Elser commented on HBASE-18843: Yeah, we removed these annotations in the last week or so. We're using equivalent ones from Yetus now. > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[
https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176797#comment-16176797
]
Ted Yu commented on HBASE-18843:
Patch v2 doesn't compile:
{code}
[ERROR]
/hbase/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/FixedRelativePathCopyListing.java:[30,46]
package org.apache.hadoop.hbase.classification does not exist
[ERROR]
/hbase/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/FixedRelativePathCopyListing.java:[54,19]
package InterfaceAudience does not exist
{code}
> Add DistCp support to incremental backup with bulk loading
> --
>
> Key: HBASE-18843
> URL: https://issues.apache.org/jira/browse/HBASE-18843
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch
>
>
> Currently, we copy bulk loaded files to backup one-by-one on a client side
> (where backup create runs). This has to be replaced with DistCp copying.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[
https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175700#comment-16175700
]
Vladimir Rodionov commented on HBASE-18843:
---
This method I need to overwrite
{code}
private Path computeSourceRootPath(FileStatus sourceStatus,
DistCpOptions options) throws IOException {
{code}
> Add DistCp support to incremental backup with bulk loading
> --
>
> Key: HBASE-18843
> URL: https://issues.apache.org/jira/browse/HBASE-18843
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch
>
>
> Currently, we copy bulk loaded files to backup one-by-one on a client side
> (where backup create runs). This has to be replaced with DistCp copying.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175679#comment-16175679 ] Ted Yu commented on HBASE-18843: Can you list which private methods you depend on ? > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[ https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175677#comment-16175677 ] Vladimir Rodionov commented on HBASE-18843: --- Created RB: https://reviews.apache.org/r/62486/ > Add DistCp support to incremental backup with bulk loading > -- > > Key: HBASE-18843 > URL: https://issues.apache.org/jira/browse/HBASE-18843 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-18843-v1.patch, HBASE-18843-v2.patch > > > Currently, we copy bulk loaded files to backup one-by-one on a client side > (where backup create runs). This has to be replaced with DistCp copying. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[
https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175669#comment-16175669
]
Vladimir Rodionov commented on HBASE-18843:
---
{quote}
Have you considered using other format which is more amiable to upgrade ?
{quote}
That is format DistCp understands and expects.
> Add DistCp support to incremental backup with bulk loading
> --
>
> Key: HBASE-18843
> URL: https://issues.apache.org/jira/browse/HBASE-18843
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-18843-v1.patch
>
>
> Currently, we copy bulk loaded files to backup one-by-one on a client side
> (where backup create runs). This has to be replaced with DistCp copying.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[
https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175665#comment-16175665
]
Vladimir Rodionov commented on HBASE-18843:
---
{quote}
Can you do some refactoring ?
{quote}
Does not seem feasible. SimpleCopyListing has some private methods I need
access to.
> Add DistCp support to incremental backup with bulk loading
> --
>
> Key: HBASE-18843
> URL: https://issues.apache.org/jira/browse/HBASE-18843
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-18843-v1.patch
>
>
> Currently, we copy bulk loaded files to backup one-by-one on a client side
> (where backup create runs). This has to be replaced with DistCp copying.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[
https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175216#comment-16175216
]
Ted Yu commented on HBASE-18843:
In this method:
{code}
+ public void doBuildListing(SequenceFile.Writer fileListWriter,
+ DistCpOptions options) throws IOException {
{code}
There is quite some duplicate code with the SimpleCopyListing class.
Can you do some refactoring ?
> Add DistCp support to incremental backup with bulk loading
> --
>
> Key: HBASE-18843
> URL: https://issues.apache.org/jira/browse/HBASE-18843
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-18843-v1.patch
>
>
> Currently, we copy bulk loaded files to backup one-by-one on a client side
> (where backup create runs). This has to be replaced with DistCp copying.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
[
https://issues.apache.org/jira/browse/HBASE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175194#comment-16175194
]
Ted Yu commented on HBASE-18843:
{code}
+public class FixedRelativePathCopyListing extends SimpleCopyListing {
{code}
Add annotation for audience.
Please put patch on review board.
{code}
+ private SequenceFile.Writer getWriter(Path pathToListFile) throws
IOException {
{code}
Have you considered using other format which is more amiable to upgrade ?
Can you add some test for this new class ?
> Add DistCp support to incremental backup with bulk loading
> --
>
> Key: HBASE-18843
> URL: https://issues.apache.org/jira/browse/HBASE-18843
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-18843-v1.patch
>
>
> Currently, we copy bulk loaded files to backup one-by-one on a client side
> (where backup create runs). This has to be replaced with DistCp copying.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
