[jira] [Commented] (HBASE-21381) Document the hadoop versions using which backup and restore feature works
[
https://issues.apache.org/jira/browse/HBASE-21381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824026#comment-16824026
]
Hudson commented on HBASE-21381:
Results for branch master
[build #954 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/954/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/954//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/954//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/954//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Document the hadoop versions using which backup and restore feature works
> -
>
> Key: HBASE-21381
> URL: https://issues.apache.org/jira/browse/HBASE-21381
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21381-1.patch, HBASE-21381-2.patch,
> HBASE-21381-3.patch
>
>
> HADOOP-15850 fixes a bug where CopyCommitter#concatFileChunks unconditionally
> tried to concatenate the files being DistCp'ed to target cluster (though the
> files are independent).
> Following is the log snippet of the failed concatenation attempt:
> {code}
> 2018-10-13 14:09:25,351 WARN [Thread-936] mapred.LocalJobRunner$Job(590):
> job_local1795473782_0004
> java.io.IOException: Inconsistent sequence file: current chunk file
> org.apache.hadoop.tools.CopyListingFileStatus@bb8826ee{hdfs://localhost:42796/user/hbase/test-data/
>
> 160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_
> length = 5100 aclEntries = null, xAttrs = null} doesnt match prior entry
> org.apache.hadoop.tools.CopyListingFileStatus@243d544d{hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-
>
> 2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_
> length = 5142 aclEntries = null, xAttrs = null}
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
> {code}
> Backup and Restore uses DistCp to transfer files between clusters.
> Without the fix from HADOOP-15850, the transfer would fail.
> This issue is to document the hadoop versions which contain HADOOP-15850 so
> that user of Backup and Restore feature knows which hadoop versions they can
> use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21381) Document the hadoop versions using which backup and restore feature works
[
https://issues.apache.org/jira/browse/HBASE-21381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16676605#comment-16676605
]
Hudson commented on HBASE-21381:
Results for branch master
[build #588 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/588/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/588//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/588//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/588//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Document the hadoop versions using which backup and restore feature works
> -
>
> Key: HBASE-21381
> URL: https://issues.apache.org/jira/browse/HBASE-21381
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21381-1.patch, HBASE-21381-2.patch,
> HBASE-21381-3.patch
>
>
> HADOOP-15850 fixes a bug where CopyCommitter#concatFileChunks unconditionally
> tried to concatenate the files being DistCp'ed to target cluster (though the
> files are independent).
> Following is the log snippet of the failed concatenation attempt:
> {code}
> 2018-10-13 14:09:25,351 WARN [Thread-936] mapred.LocalJobRunner$Job(590):
> job_local1795473782_0004
> java.io.IOException: Inconsistent sequence file: current chunk file
> org.apache.hadoop.tools.CopyListingFileStatus@bb8826ee{hdfs://localhost:42796/user/hbase/test-data/
>
> 160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_
> length = 5100 aclEntries = null, xAttrs = null} doesnt match prior entry
> org.apache.hadoop.tools.CopyListingFileStatus@243d544d{hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-
>
> 2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_
> length = 5142 aclEntries = null, xAttrs = null}
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
> {code}
> Backup and Restore uses DistCp to transfer files between clusters.
> Without the fix from HADOOP-15850, the transfer would fail.
> This issue is to document the hadoop versions which contain HADOOP-15850 so
> that user of Backup and Restore feature knows which hadoop versions they can
> use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21381) Document the hadoop versions using which backup and restore feature works
[
https://issues.apache.org/jira/browse/HBASE-21381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16676090#comment-16676090
]
liubangchen commented on HBASE-21381:
-
Not at all . If some thing I can do it just say . Thanks.
> Document the hadoop versions using which backup and restore feature works
> -
>
> Key: HBASE-21381
> URL: https://issues.apache.org/jira/browse/HBASE-21381
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21381-1.patch, HBASE-21381-2.patch,
> HBASE-21381-3.patch
>
>
> HADOOP-15850 fixes a bug where CopyCommitter#concatFileChunks unconditionally
> tried to concatenate the files being DistCp'ed to target cluster (though the
> files are independent).
> Following is the log snippet of the failed concatenation attempt:
> {code}
> 2018-10-13 14:09:25,351 WARN [Thread-936] mapred.LocalJobRunner$Job(590):
> job_local1795473782_0004
> java.io.IOException: Inconsistent sequence file: current chunk file
> org.apache.hadoop.tools.CopyListingFileStatus@bb8826ee{hdfs://localhost:42796/user/hbase/test-data/
>
> 160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_
> length = 5100 aclEntries = null, xAttrs = null} doesnt match prior entry
> org.apache.hadoop.tools.CopyListingFileStatus@243d544d{hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-
>
> 2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_
> length = 5142 aclEntries = null, xAttrs = null}
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
> {code}
> Backup and Restore uses DistCp to transfer files between clusters.
> Without the fix from HADOOP-15850, the transfer would fail.
> This issue is to document the hadoop versions which contain HADOOP-15850 so
> that user of Backup and Restore feature knows which hadoop versions they can
> use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21381) Document the hadoop versions using which backup and restore feature works
[
https://issues.apache.org/jira/browse/HBASE-21381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16676067#comment-16676067
]
Wei-Chiu Chuang commented on HBASE-21381:
-
+1 (non-binding) Thanks!
> Document the hadoop versions using which backup and restore feature works
> -
>
> Key: HBASE-21381
> URL: https://issues.apache.org/jira/browse/HBASE-21381
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-21381-1.patch, HBASE-21381-2.patch,
> HBASE-21381-3.patch
>
>
> HADOOP-15850 fixes a bug where CopyCommitter#concatFileChunks unconditionally
> tried to concatenate the files being DistCp'ed to target cluster (though the
> files are independent).
> Following is the log snippet of the failed concatenation attempt:
> {code}
> 2018-10-13 14:09:25,351 WARN [Thread-936] mapred.LocalJobRunner$Job(590):
> job_local1795473782_0004
> java.io.IOException: Inconsistent sequence file: current chunk file
> org.apache.hadoop.tools.CopyListingFileStatus@bb8826ee{hdfs://localhost:42796/user/hbase/test-data/
>
> 160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_
> length = 5100 aclEntries = null, xAttrs = null} doesnt match prior entry
> org.apache.hadoop.tools.CopyListingFileStatus@243d544d{hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-
>
> 2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_
> length = 5142 aclEntries = null, xAttrs = null}
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
> {code}
> Backup and Restore uses DistCp to transfer files between clusters.
> Without the fix from HADOOP-15850, the transfer would fail.
> This issue is to document the hadoop versions which contain HADOOP-15850 so
> that user of Backup and Restore feature knows which hadoop versions they can
> use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21381) Document the hadoop versions using which backup and restore feature works
[
https://issues.apache.org/jira/browse/HBASE-21381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16676010#comment-16676010
]
Hadoop QA commented on HBASE-21381:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 4m
56s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 4m
55s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21381 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12947001/HBASE-21381-3.patch |
| Optional Tests | dupname asflicense refguide |
| uname | Linux 8d0b2729e890 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 01603278a3 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| refguide |
https://builds.apache.org/job/PreCommit-HBASE-Build/14960/artifact/patchprocess/branch-site/book.html
|
| refguide |
https://builds.apache.org/job/PreCommit-HBASE-Build/14960/artifact/patchprocess/patch-site/book.html
|
| Max. process+thread count | 97 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14960/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Document the hadoop versions using which backup and restore feature works
> -
>
> Key: HBASE-21381
> URL: https://issues.apache.org/jira/browse/HBASE-21381
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-21381-1.patch, HBASE-21381-2.patch,
> HBASE-21381-3.patch
>
>
> HADOOP-15850 fixes a bug where CopyCommitter#concatFileChunks unconditionally
> tried to concatenate the files being DistCp'ed to target cluster (though the
> files are independent).
> Following is the log snippet of the failed concatenation attempt:
> {code}
> 2018-10-13 14:09:25,351 WARN [Thread-936] mapred.LocalJobRunner$Job(590):
> job_local1795473782_0004
> java.io.IOException: Inconsistent sequence file: current chunk file
> org.apache.hadoop.tools.CopyListingFileStatus@bb8826ee{hdfs://localhost:42796/user/hbase/test-data/
>
> 160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_
> length = 5100 aclEntries = null, xAttrs = null} doesnt match prior entry
> org.apache.hadoop.tools.CopyListingFileStatus@243d544d{hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-
>
> 2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_
> length = 5142 aclEntries = null, xAttrs = null}
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
[jira] [Commented] (HBASE-21381) Document the hadoop versions using which backup and restore feature works
[
https://issues.apache.org/jira/browse/HBASE-21381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675762#comment-16675762
]
Ted Yu commented on HBASE-21381:
liubang:
2.7.x and 2.8.y should be added as supported hadoop releases since they were
not affected by HADOOP-11794
> Document the hadoop versions using which backup and restore feature works
> -
>
> Key: HBASE-21381
> URL: https://issues.apache.org/jira/browse/HBASE-21381
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-21381-1.patch, HBASE-21381-2.patch
>
>
> HADOOP-15850 fixes a bug where CopyCommitter#concatFileChunks unconditionally
> tried to concatenate the files being DistCp'ed to target cluster (though the
> files are independent).
> Following is the log snippet of the failed concatenation attempt:
> {code}
> 2018-10-13 14:09:25,351 WARN [Thread-936] mapred.LocalJobRunner$Job(590):
> job_local1795473782_0004
> java.io.IOException: Inconsistent sequence file: current chunk file
> org.apache.hadoop.tools.CopyListingFileStatus@bb8826ee{hdfs://localhost:42796/user/hbase/test-data/
>
> 160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_
> length = 5100 aclEntries = null, xAttrs = null} doesnt match prior entry
> org.apache.hadoop.tools.CopyListingFileStatus@243d544d{hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-
>
> 2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_
> length = 5142 aclEntries = null, xAttrs = null}
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
> {code}
> Backup and Restore uses DistCp to transfer files between clusters.
> Without the fix from HADOOP-15850, the transfer would fail.
> This issue is to document the hadoop versions which contain HADOOP-15850 so
> that user of Backup and Restore feature knows which hadoop versions they can
> use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21381) Document the hadoop versions using which backup and restore feature works
[
https://issues.apache.org/jira/browse/HBASE-21381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675576#comment-16675576
]
Wei-Chiu Chuang commented on HBASE-21381:
-
Hadoop 2.7.x and 2.8.x should also work with B&R, I suppose?
> Document the hadoop versions using which backup and restore feature works
> -
>
> Key: HBASE-21381
> URL: https://issues.apache.org/jira/browse/HBASE-21381
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-21381-1.patch, HBASE-21381-2.patch
>
>
> HADOOP-15850 fixes a bug where CopyCommitter#concatFileChunks unconditionally
> tried to concatenate the files being DistCp'ed to target cluster (though the
> files are independent).
> Following is the log snippet of the failed concatenation attempt:
> {code}
> 2018-10-13 14:09:25,351 WARN [Thread-936] mapred.LocalJobRunner$Job(590):
> job_local1795473782_0004
> java.io.IOException: Inconsistent sequence file: current chunk file
> org.apache.hadoop.tools.CopyListingFileStatus@bb8826ee{hdfs://localhost:42796/user/hbase/test-data/
>
> 160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_
> length = 5100 aclEntries = null, xAttrs = null} doesnt match prior entry
> org.apache.hadoop.tools.CopyListingFileStatus@243d544d{hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-
>
> 2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_
> length = 5142 aclEntries = null, xAttrs = null}
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
> {code}
> Backup and Restore uses DistCp to transfer files between clusters.
> Without the fix from HADOOP-15850, the transfer would fail.
> This issue is to document the hadoop versions which contain HADOOP-15850 so
> that user of Backup and Restore feature knows which hadoop versions they can
> use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21381) Document the hadoop versions using which backup and restore feature works
[
https://issues.apache.org/jira/browse/HBASE-21381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675530#comment-16675530
]
Ted Yu commented on HBASE-21381:
lgtm
> Document the hadoop versions using which backup and restore feature works
> -
>
> Key: HBASE-21381
> URL: https://issues.apache.org/jira/browse/HBASE-21381
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-21381-1.patch, HBASE-21381-2.patch
>
>
> HADOOP-15850 fixes a bug where CopyCommitter#concatFileChunks unconditionally
> tried to concatenate the files being DistCp'ed to target cluster (though the
> files are independent).
> Following is the log snippet of the failed concatenation attempt:
> {code}
> 2018-10-13 14:09:25,351 WARN [Thread-936] mapred.LocalJobRunner$Job(590):
> job_local1795473782_0004
> java.io.IOException: Inconsistent sequence file: current chunk file
> org.apache.hadoop.tools.CopyListingFileStatus@bb8826ee{hdfs://localhost:42796/user/hbase/test-data/
>
> 160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_
> length = 5100 aclEntries = null, xAttrs = null} doesnt match prior entry
> org.apache.hadoop.tools.CopyListingFileStatus@243d544d{hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-
>
> 2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_
> length = 5142 aclEntries = null, xAttrs = null}
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
> {code}
> Backup and Restore uses DistCp to transfer files between clusters.
> Without the fix from HADOOP-15850, the transfer would fail.
> This issue is to document the hadoop versions which contain HADOOP-15850 so
> that user of Backup and Restore feature knows which hadoop versions they can
> use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21381) Document the hadoop versions using which backup and restore feature works
[
https://issues.apache.org/jira/browse/HBASE-21381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16674743#comment-16674743
]
Hadoop QA commented on HBASE-21381:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 5m
13s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 4m
59s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21381 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946862/HBASE-21381-2.patch |
| Optional Tests | dupname asflicense refguide |
| uname | Linux 5de67a5fa026 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ea0f1391f1 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| refguide |
https://builds.apache.org/job/PreCommit-HBASE-Build/14951/artifact/patchprocess/branch-site/book.html
|
| refguide |
https://builds.apache.org/job/PreCommit-HBASE-Build/14951/artifact/patchprocess/patch-site/book.html
|
| Max. process+thread count | 97 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14951/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Document the hadoop versions using which backup and restore feature works
> -
>
> Key: HBASE-21381
> URL: https://issues.apache.org/jira/browse/HBASE-21381
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-21381-1.patch, HBASE-21381-2.patch
>
>
> HADOOP-15850 fixes a bug where CopyCommitter#concatFileChunks unconditionally
> tried to concatenate the files being DistCp'ed to target cluster (though the
> files are independent).
> Following is the log snippet of the failed concatenation attempt:
> {code}
> 2018-10-13 14:09:25,351 WARN [Thread-936] mapred.LocalJobRunner$Job(590):
> job_local1795473782_0004
> java.io.IOException: Inconsistent sequence file: current chunk file
> org.apache.hadoop.tools.CopyListingFileStatus@bb8826ee{hdfs://localhost:42796/user/hbase/test-data/
>
> 160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_
> length = 5100 aclEntries = null, xAttrs = null} doesnt match prior entry
> org.apache.hadoop.tools.CopyListingFileStatus@243d544d{hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-
>
> 2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_
> length = 5142 aclEntries = null, xAttrs = null}
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
> at org.apache.hadoop
[jira] [Commented] (HBASE-21381) Document the hadoop versions using which backup and restore feature works
[
https://issues.apache.org/jira/browse/HBASE-21381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16674650#comment-16674650
]
Ted Yu commented on HBASE-21381:
{code}
61 * 3.2.0+
62 * 2.9.2+
63 * 3.0.4+
{code}
The hadoop versions are not sorted.
Normally it is easier to find the hadoop version the user is deploying if the
versions are in sorted order.
> Document the hadoop versions using which backup and restore feature works
> -
>
> Key: HBASE-21381
> URL: https://issues.apache.org/jira/browse/HBASE-21381
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-21381-1.patch
>
>
> HADOOP-15850 fixes a bug where CopyCommitter#concatFileChunks unconditionally
> tried to concatenate the files being DistCp'ed to target cluster (though the
> files are independent).
> Following is the log snippet of the failed concatenation attempt:
> {code}
> 2018-10-13 14:09:25,351 WARN [Thread-936] mapred.LocalJobRunner$Job(590):
> job_local1795473782_0004
> java.io.IOException: Inconsistent sequence file: current chunk file
> org.apache.hadoop.tools.CopyListingFileStatus@bb8826ee{hdfs://localhost:42796/user/hbase/test-data/
>
> 160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_
> length = 5100 aclEntries = null, xAttrs = null} doesnt match prior entry
> org.apache.hadoop.tools.CopyListingFileStatus@243d544d{hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-
>
> 2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_
> length = 5142 aclEntries = null, xAttrs = null}
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
> {code}
> Backup and Restore uses DistCp to transfer files between clusters.
> Without the fix from HADOOP-15850, the transfer would fail.
> This issue is to document the hadoop versions which contain HADOOP-15850 so
> that user of Backup and Restore feature knows which hadoop versions they can
> use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21381) Document the hadoop versions using which backup and restore feature works
[
https://issues.apache.org/jira/browse/HBASE-21381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16674645#comment-16674645
]
Hadoop QA commented on HBASE-21381:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 6m
41s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 6m
35s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21381 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946854/HBASE-21381-1.patch |
| Optional Tests | dupname asflicense refguide |
| uname | Linux 29d0bdd36b33 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ea0f1391f1 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| refguide |
https://builds.apache.org/job/PreCommit-HBASE-Build/14950/artifact/patchprocess/branch-site/book.html
|
| refguide |
https://builds.apache.org/job/PreCommit-HBASE-Build/14950/artifact/patchprocess/patch-site/book.html
|
| Max. process+thread count | 83 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14950/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Document the hadoop versions using which backup and restore feature works
> -
>
> Key: HBASE-21381
> URL: https://issues.apache.org/jira/browse/HBASE-21381
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-21381-1.patch
>
>
> HADOOP-15850 fixes a bug where CopyCommitter#concatFileChunks unconditionally
> tried to concatenate the files being DistCp'ed to target cluster (though the
> files are independent).
> Following is the log snippet of the failed concatenation attempt:
> {code}
> 2018-10-13 14:09:25,351 WARN [Thread-936] mapred.LocalJobRunner$Job(590):
> job_local1795473782_0004
> java.io.IOException: Inconsistent sequence file: current chunk file
> org.apache.hadoop.tools.CopyListingFileStatus@bb8826ee{hdfs://localhost:42796/user/hbase/test-data/
>
> 160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_
> length = 5100 aclEntries = null, xAttrs = null} doesnt match prior entry
> org.apache.hadoop.tools.CopyListingFileStatus@243d544d{hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-
>
> 2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_
> length = 5142 aclEntries = null, xAttrs = null}
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
> at org.apache.hadoop.mapred.LocalJobRunn
[jira] [Commented] (HBASE-21381) Document the hadoop versions using which backup and restore feature works
[
https://issues.apache.org/jira/browse/HBASE-21381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670811#comment-16670811
]
Ted Yu commented on HBASE-21381:
Putting the supported version under
http://hbase.apache.org/book.html#backuprestore is fine.
> Document the hadoop versions using which backup and restore feature works
> -
>
> Key: HBASE-21381
> URL: https://issues.apache.org/jira/browse/HBASE-21381
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Priority: Major
>
> HADOOP-15850 fixes a bug where CopyCommitter#concatFileChunks unconditionally
> tried to concatenate the files being DistCp'ed to target cluster (though the
> files are independent).
> Following is the log snippet of the failed concatenation attempt:
> {code}
> 2018-10-13 14:09:25,351 WARN [Thread-936] mapred.LocalJobRunner$Job(590):
> job_local1795473782_0004
> java.io.IOException: Inconsistent sequence file: current chunk file
> org.apache.hadoop.tools.CopyListingFileStatus@bb8826ee{hdfs://localhost:42796/user/hbase/test-data/
>
> 160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_
> length = 5100 aclEntries = null, xAttrs = null} doesnt match prior entry
> org.apache.hadoop.tools.CopyListingFileStatus@243d544d{hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-
>
> 2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_
> length = 5142 aclEntries = null, xAttrs = null}
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
> {code}
> Backup and Restore uses DistCp to transfer files between clusters.
> Without the fix from HADOOP-15850, the transfer would fail.
> This issue is to document the hadoop versions which contain HADOOP-15850 so
> that user of Backup and Restore feature knows which hadoop versions they can
> use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21381) Document the hadoop versions using which backup and restore feature works
[
https://issues.apache.org/jira/browse/HBASE-21381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662653#comment-16662653
]
Sean Busbey commented on HBASE-21381:
-
probably should be another blurb under the Hadoop matrix here:
[http://hbase.apache.org/book.html#hadoop]
bit awkward since B&R isn't in a release line.
maybe a call out in the B&R section instead? I can't think of another section
that has hadoop specific details like this in it:
http://hbase.apache.org/book.html#backuprestore
> Document the hadoop versions using which backup and restore feature works
> -
>
> Key: HBASE-21381
> URL: https://issues.apache.org/jira/browse/HBASE-21381
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Priority: Major
>
> HADOOP-15850 fixes a bug where CopyCommitter#concatFileChunks unconditionally
> tried to concatenate the files being DistCp'ed to target cluster (though the
> files are independent).
> Following is the log snippet of the failed concatenation attempt:
> {code}
> 2018-10-13 14:09:25,351 WARN [Thread-936] mapred.LocalJobRunner$Job(590):
> job_local1795473782_0004
> java.io.IOException: Inconsistent sequence file: current chunk file
> org.apache.hadoop.tools.CopyListingFileStatus@bb8826ee{hdfs://localhost:42796/user/hbase/test-data/
>
> 160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_
> length = 5100 aclEntries = null, xAttrs = null} doesnt match prior entry
> org.apache.hadoop.tools.CopyListingFileStatus@243d544d{hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-
>
> 2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_
> length = 5142 aclEntries = null, xAttrs = null}
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
> {code}
> Backup and Restore uses DistCp to transfer files between clusters.
> Without the fix from HADOOP-15850, the transfer would fail.
> This issue is to document the hadoop versions which contain HADOOP-15850 so
> that user of Backup and Restore feature knows which hadoop versions they can
> use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21381) Document the hadoop versions using which backup and restore feature works
[
https://issues.apache.org/jira/browse/HBASE-21381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662458#comment-16662458
]
Wei-Chiu Chuang commented on HBASE-21381:
-
Since it was broken by HADOOP-11794, 2.9 and 3.x releases.
We're preparing the 3.2.0 release and probably should port this fix in 3.2.0.
> Document the hadoop versions using which backup and restore feature works
> -
>
> Key: HBASE-21381
> URL: https://issues.apache.org/jira/browse/HBASE-21381
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Priority: Major
>
> HADOOP-15850 fixes a bug where CopyCommitter#concatFileChunks unconditionally
> tried to concatenate the files being DistCp'ed to target cluster (though the
> files are independent).
> Following is the log snippet of the failed concatenation attempt:
> {code}
> 2018-10-13 14:09:25,351 WARN [Thread-936] mapred.LocalJobRunner$Job(590):
> job_local1795473782_0004
> java.io.IOException: Inconsistent sequence file: current chunk file
> org.apache.hadoop.tools.CopyListingFileStatus@bb8826ee{hdfs://localhost:42796/user/hbase/test-data/
>
> 160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_
> length = 5100 aclEntries = null, xAttrs = null} doesnt match prior entry
> org.apache.hadoop.tools.CopyListingFileStatus@243d544d{hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-
>
> 2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_
> length = 5142 aclEntries = null, xAttrs = null}
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
> {code}
> Backup and Restore uses DistCp to transfer files between clusters.
> Without the fix from HADOOP-15850, the transfer would fail.
> This issue is to document the hadoop versions which contain HADOOP-15850 so
> that user of Backup and Restore feature knows which hadoop versions they can
> use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
