[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs in HDFS"
[ https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909651#comment-16909651 ] Wei-Chiu Chuang commented on HDFS-14147: [~vrushalic] this looks good. would you still like to push this forward? > Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs > in HDFS" > > > Key: HDFS-14147 > URL: https://issues.apache.org/jira/browse/HDFS-14147 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, hdfs >Affects Versions: 2.9.0, 2.9.1, 2.9.2 >Reporter: Yan >Assignee: Yan >Priority: Major > Attachments: HDFS-14147-branch-2.9-001.patch, > HDFS-14147-branch-2.9-001.patch, HDFS-14147.pdf > > > HDFS-13056, Expose file-level composite CRCs in HDFS which are comparable > across different instances/layouts, is a significant feature for storage > agnostic CRC comparisons between HDFS and cloud object stores such as S3 and > GCS. With the extensively installed base of Hadoop 2, it should make a lot of > sense to have the feature in Hadoop 2. > The plan is to start with the backporting to 2.9, followed by 2.8 and 2.7 in > that order. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs in HDFS"
[
https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896810#comment-16896810
]
Hadoop QA commented on HDFS-14147:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 24m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.9 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
17s{color} | {color:green} branch-2.9 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
12s{color} | {color:green} branch-2.9 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
6s{color} | {color:green} branch-2.9 passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
17s{color} | {color:green} branch-2.9 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
33s{color} | {color:green} branch-2.9 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
33s{color} | {color:green} branch-2.9 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
40s{color} | {color:green} branch-2.9 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
53s{color} | {color:green} branch-2.9 passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
2s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 15m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 15m 2s{color}
| {color:red} root-jdk1.7.0_95 with JDK v1.7.0_95 generated 3 new + 1443
unchanged - 3 fixed = 1446 total (was 1446) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
38s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 13m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
38s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 15s{color} | {color:orange} root: The patch generated 16 new + 880 unchanged
- 7 fixed = 896 total (was 887) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
55s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
29s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
44s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
34s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 25s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
|
[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs in HDFS"
[ https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740049#comment-16740049 ] Yan commented on HDFS-14147: Timeout after 5 hours again with the following errors: cd /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs /opt/maven/bin/mvn --batch-mode -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build@2/yetus-m2/hadoop-branch-2.9-patch-0 -Ptest-patch -Pparallel-tests -P!shelltest -Pnative -Drequire.fuse -Drequire.openssl -Drequire.snappy -Drequire.valgrind -Drequire.test.libhadoop -Pyarn-ui clean test -fae > /testptch/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt 2>&1 Build timed out (after 300 minutes). Marking the build as aborted. Are there any environmental issues recently? Any other build process has succeeded recently? > Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs > in HDFS" > > > Key: HDFS-14147 > URL: https://issues.apache.org/jira/browse/HDFS-14147 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, hdfs >Affects Versions: 2.9.0, 2.9.1, 2.9.2 >Reporter: Yan >Priority: Major > Attachments: HDFS-14147-branch-2.9-001.patch, > HDFS-14147-branch-2.9-001.patch, HDFS-14147.pdf > > > HDFS-13056, Expose file-level composite CRCs in HDFS which are comparable > across different instances/layouts, is a significant feature for storage > agnostic CRC comparisons between HDFS and cloud object stores such as S3 and > GCS. With the extensively installed base of Hadoop 2, it should make a lot of > sense to have the feature in Hadoop 2. > The plan is to start with the backporting to 2.9, followed by 2.8 and 2.7 in > that order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs in HDFS"
[ https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737809#comment-16737809 ] Vrushali C commented on HDFS-14147: --- Thanks [~yzhou2001]! That helps set the context for me. I have re-triggered the build for this patch at https://builds.apache.org/job/PreCommit-HDFS-Build/25936/console For some reason the previous build had timed out: https://builds.apache.org/job/PreCommit-HDFS-Build/25923/console > Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs > in HDFS" > > > Key: HDFS-14147 > URL: https://issues.apache.org/jira/browse/HDFS-14147 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, hdfs >Affects Versions: 2.9.0, 2.9.1, 2.9.2 >Reporter: Yan >Priority: Major > Attachments: HDFS-14147-branch-2.9-001.patch, > HDFS-14147-branch-2.9-001.patch, HDFS-14147.pdf > > > HDFS-13056, Expose file-level composite CRCs in HDFS which are comparable > across different instances/layouts, is a significant feature for storage > agnostic CRC comparisons between HDFS and cloud object stores such as S3 and > GCS. With the extensively installed base of Hadoop 2, it should make a lot of > sense to have the feature in Hadoop 2. > The plan is to start with the backporting to 2.9, followed by 2.8 and 2.7 in > that order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs in HDFS"
[ https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737739#comment-16737739 ] Yan commented on HDFS-14147: Thanks [~vrushalic] for the hints to trigger jenkins. I have tried various options as suggested in the past few days. None seemed successful. On your questions on the feature, my answers are as follows: 1) The feature is not allowing the comparison. One can always compare the checksums. But without this feature, the comparison won't make sense between HDFS files of different block sizes/chunk sizes, or between a HDFS file and one on a different storage systems, etc; 2) The feature is runtime behavior on-the-fly and has no persistent impact. And the default HDFS client behavior is the old checksum computation approach. So there are no version compatibility issues between a new HDFS software against existing HDFS persistent data. 3) Again by default HDFS client uses the old "MD5MD5CRC" algorithm to compute the HDFS file checksum; the new "composite crc" algorithm has to be used explicitly with the dfs.checksum.combine.mode configuration flag being set to COMPOSITE_CRC. > Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs > in HDFS" > > > Key: HDFS-14147 > URL: https://issues.apache.org/jira/browse/HDFS-14147 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, hdfs >Affects Versions: 2.9.0, 2.9.1, 2.9.2 >Reporter: Yan >Priority: Major > Attachments: HDFS-14147-branch-2.9-001.patch, > HDFS-14147-branch-2.9-001.patch, HDFS-14147.pdf > > > HDFS-13056, Expose file-level composite CRCs in HDFS which are comparable > across different instances/layouts, is a significant feature for storage > agnostic CRC comparisons between HDFS and cloud object stores such as S3 and > GCS. With the extensively installed base of Hadoop 2, it should make a lot of > sense to have the feature in Hadoop 2. > The plan is to start with the backporting to 2.9, followed by 2.8 and 2.7 in > that order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs in HDFS"
[ https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737656#comment-16737656 ] Vrushali C commented on HDFS-14147: --- Thanks for the backported patch [~yzhou2001] ! I am not an HDFS expert in any way but I would like to ask for a few clarifications which may make it easier to decide about the backport of this feature from trunk to branch-2. IIUC this patch adds a new FileChecksum type (called "COMPOSITE-CRC") which allows for comparison between HDFS and other external storage systems. Is that understanding correct? - Is this an incompatible change on the datanode metadata? Or elsewhere on the cluster as well? - If I have an existing 2.9.0 cluster and say I release a newer 2.9.x which contains this new checksum type, will it in anyway affect the older, existing clients or they can continue to run as before if they do not want to make use of this feature. - Is this reversible? For example, if I want to downgrade the hadoop version (for unrelated reasons), what is the impact of having had this feature and then going back to an older version which does not have this capability? Does datanode metadata need to be rewritten? - Is this on or off by default > Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs > in HDFS" > > > Key: HDFS-14147 > URL: https://issues.apache.org/jira/browse/HDFS-14147 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, hdfs >Affects Versions: 2.9.0, 2.9.1, 2.9.2 >Reporter: Yan >Priority: Major > Attachments: HDFS-14147-branch-2.9-001.patch, > HDFS-14147-branch-2.9-001.patch, HDFS-14147.pdf > > > HDFS-13056, Expose file-level composite CRCs in HDFS which are comparable > across different instances/layouts, is a significant feature for storage > agnostic CRC comparisons between HDFS and cloud object stores such as S3 and > GCS. With the extensively installed base of Hadoop 2, it should make a lot of > sense to have the feature in Hadoop 2. > The plan is to start with the backporting to 2.9, followed by 2.8 and 2.7 in > that order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs in HDFS"
[ https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734640#comment-16734640 ] Yan commented on HDFS-14147: Hi [~ste...@apache.org], could you trigger another test run from [~hadoopqa] ? Thanks. > Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs > in HDFS" > > > Key: HDFS-14147 > URL: https://issues.apache.org/jira/browse/HDFS-14147 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, hdfs >Affects Versions: 2.9.0, 2.9.1, 2.9.2 >Reporter: Yan >Priority: Major > Attachments: HDFS-14147-branch-2.9-001.patch, HDFS-14147.pdf > > > HDFS-13056, Expose file-level composite CRCs in HDFS which are comparable > across different instances/layouts, is a significant feature for storage > agnostic CRC comparisons between HDFS and cloud object stores such as S3 and > GCS. With the extensively installed base of Hadoop 2, it should make a lot of > sense to have the feature in Hadoop 2. > The plan is to start with the backporting to 2.9, followed by 2.8 and 2.7 in > that order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs in HDFS"
[ https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734618#comment-16734618 ] Steve Loughran commented on HDFS-14147: --- hit cancel, reattach the existing patch (same name is fine), hit submit patch again > Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs > in HDFS" > > > Key: HDFS-14147 > URL: https://issues.apache.org/jira/browse/HDFS-14147 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, hdfs >Affects Versions: 2.9.0, 2.9.1, 2.9.2 >Reporter: Yan >Priority: Major > Attachments: HDFS-14147-branch-2.9-001.patch, HDFS-14147.pdf > > > HDFS-13056, Expose file-level composite CRCs in HDFS which are comparable > across different instances/layouts, is a significant feature for storage > agnostic CRC comparisons between HDFS and cloud object stores such as S3 and > GCS. With the extensively installed base of Hadoop 2, it should make a lot of > sense to have the feature in Hadoop 2. > The plan is to start with the backporting to 2.9, followed by 2.8 and 2.7 in > that order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs in HDFS"
[ https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734619#comment-16734619 ] Steve Loughran commented on HDFS-14147: --- no, don't worry about the reattach, you've uploaded the new patch. Just cancel -> submit > Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs > in HDFS" > > > Key: HDFS-14147 > URL: https://issues.apache.org/jira/browse/HDFS-14147 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, hdfs >Affects Versions: 2.9.0, 2.9.1, 2.9.2 >Reporter: Yan >Priority: Major > Attachments: HDFS-14147-branch-2.9-001.patch, HDFS-14147.pdf > > > HDFS-13056, Expose file-level composite CRCs in HDFS which are comparable > across different instances/layouts, is a significant feature for storage > agnostic CRC comparisons between HDFS and cloud object stores such as S3 and > GCS. With the extensively installed base of Hadoop 2, it should make a lot of > sense to have the feature in Hadoop 2. > The plan is to start with the backporting to 2.9, followed by 2.8 and 2.7 in > that order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs in HDFS"
[ https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734441#comment-16734441 ] Yan commented on HDFS-14147: [~hadoopqa] please start off another test run. > Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs > in HDFS" > > > Key: HDFS-14147 > URL: https://issues.apache.org/jira/browse/HDFS-14147 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, hdfs >Affects Versions: 2.9.0, 2.9.1, 2.9.2 >Reporter: Yan >Priority: Major > Attachments: HDFS-14147-branch-2.9-001.patch, HDFS-14147.pdf > > > HDFS-13056, Expose file-level composite CRCs in HDFS which are comparable > across different instances/layouts, is a significant feature for storage > agnostic CRC comparisons between HDFS and cloud object stores such as S3 and > GCS. With the extensively installed base of Hadoop 2, it should make a lot of > sense to have the feature in Hadoop 2. > The plan is to start with the backporting to 2.9, followed by 2.8 and 2.7 in > that order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs in HDFS"
[ https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733482#comment-16733482 ] Yan commented on HDFS-14147: Renamed. Thanks [~ste...@apache.org] > Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs > in HDFS" > > > Key: HDFS-14147 > URL: https://issues.apache.org/jira/browse/HDFS-14147 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, hdfs >Affects Versions: 2.9.0, 2.9.1, 2.9.2 >Reporter: Yan >Priority: Major > Attachments: HDFS-14147-branch-2.9-001.patch, HDFS-14147.pdf > > > HDFS-13056, Expose file-level composite CRCs in HDFS which are comparable > across different instances/layouts, is a significant feature for storage > agnostic CRC comparisons between HDFS and cloud object stores such as S3 and > GCS. With the extensively installed base of Hadoop 2, it should make a lot of > sense to have the feature in Hadoop 2. > The plan is to start with the backporting to 2.9, followed by 2.8 and 2.7 in > that order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs in HDFS"
[ https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733469#comment-16733469 ] Steve Loughran commented on HDFS-14147: --- you need to name your patch for that; the hyphens are used to split up HDFS-14147-branch-2.9-001.patch > Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs > in HDFS" > > > Key: HDFS-14147 > URL: https://issues.apache.org/jira/browse/HDFS-14147 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, hdfs >Affects Versions: 2.9.0, 2.9.1, 2.9.2 >Reporter: Yan >Priority: Major > Attachments: HDFS-14147-branch-2.9.v1.patch, HDFS-14147.pdf > > > HDFS-13056, Expose file-level composite CRCs in HDFS which are comparable > across different instances/layouts, is a significant feature for storage > agnostic CRC comparisons between HDFS and cloud object stores such as S3 and > GCS. With the extensively installed base of Hadoop 2, it should make a lot of > sense to have the feature in Hadoop 2. > The plan is to start with the backporting to 2.9, followed by 2.8 and 2.7 in > that order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs in HDFS"
[ https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733435#comment-16733435 ] Yan commented on HDFS-14147: The patch is based on branch-2.9, not the trunk. > Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs > in HDFS" > > > Key: HDFS-14147 > URL: https://issues.apache.org/jira/browse/HDFS-14147 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, hdfs >Affects Versions: 2.9.0, 2.9.1, 2.9.2 >Reporter: Yan >Priority: Major > Attachments: HDFS-14147-branch-2.9.v1.patch, HDFS-14147.pdf > > > HDFS-13056, Expose file-level composite CRCs in HDFS which are comparable > across different instances/layouts, is a significant feature for storage > agnostic CRC comparisons between HDFS and cloud object stores such as S3 and > GCS. With the extensively installed base of Hadoop 2, it should make a lot of > sense to have the feature in Hadoop 2. > The plan is to start with the backporting to 2.9, followed by 2.8 and 2.7 in > that order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs in HDFS"
[ https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733408#comment-16733408 ] Steve Loughran commented on HDFS-14147: --- BTW * I'm not competent enough in the HDFS codebase to review those bits -1 to any backport to 2.7 & 2.8. Those are stable, which remain stable by not adding big changes to them. 2.7.x especially: security patches and major bugs only > Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs > in HDFS" > > > Key: HDFS-14147 > URL: https://issues.apache.org/jira/browse/HDFS-14147 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, hdfs >Affects Versions: 2.9.0, 2.9.1, 2.9.2 >Reporter: Yan >Priority: Major > Attachments: HDFS-14147-branch-2.9.v1.patch, HDFS-14147.pdf > > > HDFS-13056, Expose file-level composite CRCs in HDFS which are comparable > across different instances/layouts, is a significant feature for storage > agnostic CRC comparisons between HDFS and cloud object stores such as S3 and > GCS. With the extensively installed base of Hadoop 2, it should make a lot of > sense to have the feature in Hadoop 2. > The plan is to start with the backporting to 2.9, followed by 2.8 and 2.7 in > that order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs in HDFS"
[
https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733409#comment-16733409
]
Hadoop QA commented on HDFS-14147:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HDFS-14147 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-14147 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25906/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Backport of HDFS-13056 to the 2.9 branch: "Expose file-level composite CRCs
> in HDFS"
>
>
> Key: HDFS-14147
> URL: https://issues.apache.org/jira/browse/HDFS-14147
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, distcp, hdfs
>Affects Versions: 2.9.0, 2.9.1, 2.9.2
>Reporter: Yan
>Priority: Major
> Attachments: HDFS-14147-branch-2.9.v1.patch, HDFS-14147.pdf
>
>
> HDFS-13056, Expose file-level composite CRCs in HDFS which are comparable
> across different instances/layouts, is a significant feature for storage
> agnostic CRC comparisons between HDFS and cloud object stores such as S3 and
> GCS. With the extensively installed base of Hadoop 2, it should make a lot of
> sense to have the feature in Hadoop 2.
> The plan is to start with the backporting to 2.9, followed by 2.8 and 2.7 in
> that order.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch
[ https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733249#comment-16733249 ] Yan commented on HDFS-14147: Hi community, wondering any chance to get the patch reviewed and committed soon? Thanks. > Backport of HDFS-13056 to the 2.9 branch > > > Key: HDFS-14147 > URL: https://issues.apache.org/jira/browse/HDFS-14147 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, hdfs >Affects Versions: 2.9.0, 2.9.1, 2.9.2 >Reporter: Yan >Priority: Major > Attachments: HDFS-14147-branch-2.9.v1.patch, HDFS-14147.pdf > > > HDFS-13056, Expose file-level composite CRCs in HDFS which are comparable > across different instances/layouts, is a significant feature for storage > agnostic CRC comparisons between HDFS and cloud object stores such as S3 and > GCS. With the extensively installed base of Hadoop 2, it should make a lot of > sense to have the feature in Hadoop 2. > The plan is to start with the backporting to 2.9, followed by 2.8 and 2.7 in > that order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch
[ https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722014#comment-16722014 ] Dennis Huo commented on HDFS-14147: --- The branch-2.8 confusion might be because I had started with some prototype branch-2.8 patches in HDFS-13056 but I never got around to cleaning it up and refactoring for actually committing into 2.8. Yan's dependency analysis looks good to me and should allow a clean merge into 2.9, 2.8, and 2.7. > Backport of HDFS-13056 to the 2.9 branch > > > Key: HDFS-14147 > URL: https://issues.apache.org/jira/browse/HDFS-14147 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, hdfs >Affects Versions: 2.9.0, 2.9.1, 2.9.2 >Reporter: Yan >Priority: Major > Attachments: HDFS-14147-branch-2.9.v1.patch, HDFS-14147.pdf > > > HDFS-13056, Expose file-level composite CRCs in HDFS which are comparable > across different instances/layouts, is a significant feature for storage > agnostic CRC comparisons between HDFS and cloud object stores such as S3 and > GCS. With the extensively installed base of Hadoop 2, it should make a lot of > sense to have the feature in Hadoop 2. > The plan is to start with the backporting to 2.9, followed by 2.8 and 2.7 in > that order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch
[ https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721711#comment-16721711 ] Steve Loughran commented on HDFS-14147: --- FWIW, doesn't make any different for S3. That only has file level checksums, and when you turn it on it breaks all distcp runs...which is why its so very optional > Backport of HDFS-13056 to the 2.9 branch > > > Key: HDFS-14147 > URL: https://issues.apache.org/jira/browse/HDFS-14147 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, hdfs >Affects Versions: 2.9.0, 2.9.1, 2.9.2 >Reporter: Yan >Priority: Major > Attachments: HDFS-14147-branch-2.9.v1.patch, HDFS-14147.pdf > > > HDFS-13056, Expose file-level composite CRCs in HDFS which are comparable > across different instances/layouts, is a significant feature for storage > agnostic CRC comparisons between HDFS and cloud object stores such as S3 and > GCS. With the extensively installed base of Hadoop 2, it should make a lot of > sense to have the feature in Hadoop 2. > The plan is to start with the backporting to 2.9, followed by 2.8 and 2.7 in > that order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch
[ https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720676#comment-16720676 ] Giovanni Matteo Fumarola commented on HDFS-14147: - cc. [~dennishuo] , [~xiaochen] , [~ste...@apache.org] > Backport of HDFS-13056 to the 2.9 branch > > > Key: HDFS-14147 > URL: https://issues.apache.org/jira/browse/HDFS-14147 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, hdfs >Affects Versions: 2.9.0, 2.9.1, 2.9.2 >Reporter: Yan >Priority: Major > Attachments: HDFS-14147-branch-2.9.v1.patch, HDFS-14147.pdf > > > HDFS-13056, Expose file-level composite CRCs in HDFS which are comparable > across different instances/layouts, is a significant feature for storage > agnostic CRC comparisons between HDFS and cloud object stores such as S3 and > GCS. With the extensively installed base of Hadoop 2, it should make a lot of > sense to have the feature in Hadoop 2. > The plan is to start with the backporting to 2.9, followed by 2.8 and 2.7 in > that order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch
[ https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719636#comment-16719636 ] Yan commented on HDFS-14147: [~giovanni.fumarola] I have updated the patch file. However I can't find the porting of HDFS-13056 to branch-2.8. Do you have more pointers such as Jira or commit hash so I can check? Thanks. > Backport of HDFS-13056 to the 2.9 branch > > > Key: HDFS-14147 > URL: https://issues.apache.org/jira/browse/HDFS-14147 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, hdfs >Affects Versions: 2.9.0, 2.9.1, 2.9.2 >Reporter: Yan >Priority: Major > Attachments: HDFS-14147-branch-2.9.v1.patch > > > HDFS-13056, Expose file-level composite CRCs in HDFS which are comparable > across different instances/layouts, is a significant feature for storage > agnostic CRC comparisons between HDFS and cloud object stores such as S3 and > GCS. With the extensively installed base of Hadoop 2, it should make a lot of > sense to have the feature in Hadoop 2. > The plan is to start with the backporting to 2.9, followed by 2.8 and 2.7 in > that order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch
[ https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719608#comment-16719608 ] Giovanni Matteo Fumarola commented on HDFS-14147: - Thanks [~yzhou2001] . Can you rename the patch: HDFS-14147-branch-2.9.v1.patch? By doing that Yetus will run the patch in branch-2.9. > Backport of HDFS-13056 to the 2.9 branch > > > Key: HDFS-14147 > URL: https://issues.apache.org/jira/browse/HDFS-14147 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, hdfs >Affects Versions: 2.9.0, 2.9.1, 2.9.2 >Reporter: Yan >Priority: Major > Attachments: HDFS-2.9.patch > > > HDFS-13056, Expose file-level composite CRCs in HDFS which are comparable > across different instances/layouts, is a significant feature for storage > agnostic CRC comparisons between HDFS and cloud object stores such as S3 and > GCS. With the extensively installed base of Hadoop 2, it should make a lot of > sense to have the feature in Hadoop 2. > The plan is to start with the backporting to 2.9, followed by 2.8 and 2.7 in > that order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14147) Backport of HDFS-13056 to the 2.9 branch
[ https://issues.apache.org/jira/browse/HDFS-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719602#comment-16719602 ] ASF GitHub Bot commented on HDFS-14147: --- GitHub user yzhou2001 opened a pull request: https://github.com/apache/hadoop/pull/446 HDFS-14147: Back port of HDFS-13056 to the 2.9 branch You can merge this pull request into a Git repository by running: $ git pull https://github.com/yzhou2001/hadoop branch-2.9 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/446.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #446 commit e5c3dda5cdfa376ea2ad3d9284f2f9b224ccd1df Author: yzhou2001 Date: 2018-12-13T00:09:46Z Back port of HDFS-13056 to the 2.9 branch > Backport of HDFS-13056 to the 2.9 branch > > > Key: HDFS-14147 > URL: https://issues.apache.org/jira/browse/HDFS-14147 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, hdfs >Affects Versions: 2.9.0, 2.9.1, 2.9.2 >Reporter: Yan >Priority: Major > > HDFS-13056, Expose file-level composite CRCs in HDFS which are comparable > across different instances/layouts, is a significant feature for storage > agnostic CRC comparisons between HDFS and cloud object stores such as S3 and > GCS. With the extensively installed base of Hadoop 2, it should make a lot of > sense to have the feature in Hadoop 2. > The plan is to start with the backporting to 2.9, followed by 2.8 and 2.7 in > that order. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
