[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[ https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095907#comment-15095907 ] Thomas Demoor commented on HADOOP-11262: Thank you for supporting this feature and for your reviews, advice and help [~eddyxu], [~cnauroth], [~ste...@apache.org], [~mackrorysd]. Ping [~jghoman]: you showed initial interest in this, simply notifying you that it's been merged in. > Enable YARN to use S3A > --- > > Key: HADOOP-11262 > URL: https://issues.apache.org/jira/browse/HADOOP-11262 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Thomas Demoor >Assignee: Pieter Reuse > Labels: amazon, s3 > Fix For: 2.8.0 > > Attachments: HADOOP-11262-10.patch, HADOOP-11262-2.patch, > HADOOP-11262-3.patch, HADOOP-11262-4.patch, HADOOP-11262-5.patch, > HADOOP-11262-6.patch, HADOOP-11262-7.patch, HADOOP-11262-8.patch, > HADOOP-11262-9.patch, HADOOP-11262.patch > > > Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[ https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15094897#comment-15094897 ] Hudson commented on HADOOP-11262: - FAILURE: Integrated in Hadoop-trunk-Commit #9097 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9097/]) HADOOP-11262. Enable YARN to use S3A. (Pieter Reuse via lei) (lei: rev 126705f67eaa6d866f7572fbddf133c5d7552353) * hadoop-tools/hadoop-aws/src/test/resources/core-site.xml * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/fileContext/TestS3AFileContextURI.java * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/yarn/TestS3AMiniYarnCluster.java * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/fileContext/TestS3AFileContext.java * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java * hadoop-common-project/hadoop-common/src/main/resources/core-default.xml * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/yarn/TestS3A.java * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/fileContext/TestS3AFileContextUtil.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextMainOperationsBaseTest.java * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/fileContext/TestS3AFileContextCreateMkdir.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md * hadoop-tools/hadoop-aws/pom.xml * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/fileContext/TestS3AFileContextMainOperations.java * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/fileContext/TestS3AFileContextStatistics.java * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3A.java > Enable YARN to use S3A > --- > > Key: HADOOP-11262 > URL: https://issues.apache.org/jira/browse/HADOOP-11262 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Thomas Demoor >Assignee: Pieter Reuse > Labels: amazon, s3 > Fix For: 2.8.0 > > Attachments: HADOOP-11262-10.patch, HADOOP-11262-2.patch, > HADOOP-11262-3.patch, HADOOP-11262-4.patch, HADOOP-11262-5.patch, > HADOOP-11262-6.patch, HADOOP-11262-7.patch, HADOOP-11262-8.patch, > HADOOP-11262-9.patch, HADOOP-11262.patch > > > Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[ https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15094909#comment-15094909 ] Chris Nauroth commented on HADOOP-11262: [~eddyxu], thank you for taking care of the commit. > Enable YARN to use S3A > --- > > Key: HADOOP-11262 > URL: https://issues.apache.org/jira/browse/HADOOP-11262 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Thomas Demoor >Assignee: Pieter Reuse > Labels: amazon, s3 > Fix For: 2.8.0 > > Attachments: HADOOP-11262-10.patch, HADOOP-11262-2.patch, > HADOOP-11262-3.patch, HADOOP-11262-4.patch, HADOOP-11262-5.patch, > HADOOP-11262-6.patch, HADOOP-11262-7.patch, HADOOP-11262-8.patch, > HADOOP-11262-9.patch, HADOOP-11262.patch > > > Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[
https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15091745#comment-15091745
]
Hadoop QA commented on HADOOP-11262:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
0s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 55s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 28s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 27s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 49s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 29s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 56s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 13s
{color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 38s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s
{color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 59s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.metrics2.impl.TestGangliaMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[
https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15091278#comment-15091278
]
Chris Nauroth commented on HADOOP-11262:
[~PieterReuse], thanks for diving deep into the practical usage of mtime in
that last response. I agree with the trade-off. Let's proceed with this.
I see just one more test that needs a {{@Ignore}} annotation:
{{TestS3AFileContextMainOperations#testSetVerifyChecksum}}. I'll be +1 after
that.
> Enable YARN to use S3A
> ---
>
> Key: HADOOP-11262
> URL: https://issues.apache.org/jira/browse/HADOOP-11262
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Thomas Demoor
>Assignee: Pieter Reuse
> Labels: amazon, s3
> Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch,
> HADOOP-11262-4.patch, HADOOP-11262-5.patch, HADOOP-11262-6.patch,
> HADOOP-11262-7.patch, HADOOP-11262-8.patch, HADOOP-11262-9.patch,
> HADOOP-11262.patch
>
>
> Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[
https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15091284#comment-15091284
]
Hadoop QA commented on HADOOP-11262:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 36s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 16s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 26s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 18s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 46s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 12s
{color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 18s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s
{color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 11s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.metrics2.impl.TestGangliaMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[ https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089720#comment-15089720 ] Lei (Eddy) Xu commented on HADOOP-11262: [~ste...@apache.org] Sure, will do. I pinged [~Pieter Reuse] and he said there will be a new patch to address Chirs' comments. I will commit after that. > Enable YARN to use S3A > --- > > Key: HADOOP-11262 > URL: https://issues.apache.org/jira/browse/HADOOP-11262 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Thomas Demoor >Assignee: Pieter Reuse > Labels: amazon, s3 > Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch, > HADOOP-11262-4.patch, HADOOP-11262-5.patch, HADOOP-11262-6.patch, > HADOOP-11262-7.patch, HADOOP-11262-8.patch, HADOOP-11262.patch > > > Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[ https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089423#comment-15089423 ] Steve Loughran commented on HADOOP-11262: - actually, cut the copyright 2015 line entirely. It goes into NOTICE.TXT and is updated in one place only. thanks > Enable YARN to use S3A > --- > > Key: HADOOP-11262 > URL: https://issues.apache.org/jira/browse/HADOOP-11262 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Thomas Demoor >Assignee: Pieter Reuse > Labels: amazon, s3 > Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch, > HADOOP-11262-4.patch, HADOOP-11262-5.patch, HADOOP-11262-6.patch, > HADOOP-11262-7.patch, HADOOP-11262-8.patch, HADOOP-11262.patch > > > Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[ https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082011#comment-15082011 ] Lei (Eddy) Xu commented on HADOOP-11262: +1. The last patch looks very good to me. Thanks for the work, [~PieterReuse] and [~Thomas Demoor]. And thanks for the reviews, [~mackrorysd]. I will wait for [~cnauroth]'s and [~ste...@apache.org]'s opinions and will commit it tomorrow by the EOD if they are OK with this patch. Thanks guys. > Enable YARN to use S3A > --- > > Key: HADOOP-11262 > URL: https://issues.apache.org/jira/browse/HADOOP-11262 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Thomas Demoor >Assignee: Pieter Reuse > Labels: amazon, s3 > Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch, > HADOOP-11262-4.patch, HADOOP-11262-5.patch, HADOOP-11262-6.patch, > HADOOP-11262-7.patch, HADOOP-11262-8.patch, HADOOP-11262.patch > > > Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[ https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082063#comment-15082063 ] Steve Loughran commented on HADOOP-11262: - LGTM, +1 Lei -remember to bump up the year in the new copyright headers to 2016. > Enable YARN to use S3A > --- > > Key: HADOOP-11262 > URL: https://issues.apache.org/jira/browse/HADOOP-11262 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Thomas Demoor >Assignee: Pieter Reuse > Labels: amazon, s3 > Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch, > HADOOP-11262-4.patch, HADOOP-11262-5.patch, HADOOP-11262-6.patch, > HADOOP-11262-7.patch, HADOOP-11262-8.patch, HADOOP-11262.patch > > > Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[
https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082104#comment-15082104
]
Chris Nauroth commented on HADOOP-11262:
Patch v8 looks great overall. Thanks, everyone! I have just one question and
one minor nitpick.
{code}
@Override
public long getModificationTime(){
if(isDirectory()){
return System.currentTimeMillis();
} else {
return super.getModificationTime();
}
}
{code}
YARN resource localization includes a check of the modification time against
the source file system. (See {{FSDownload#copy}}.) I believe the intent is to
ensure that the state of the resources at application submission time remains
the same at the time of launching the application's containers. Has anyone
tested resource localization sourced from S3A? I expect an ever-changing
modification time like this would trigger an error.
I see you're using the strategy of overriding some irrelevant/unsupported test
methods to be no-ops in the subclasses. Could you please put the {{@Ignore}}
annotation on these?
> Enable YARN to use S3A
> ---
>
> Key: HADOOP-11262
> URL: https://issues.apache.org/jira/browse/HADOOP-11262
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Thomas Demoor
>Assignee: Pieter Reuse
> Labels: amazon, s3
> Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch,
> HADOOP-11262-4.patch, HADOOP-11262-5.patch, HADOOP-11262-6.patch,
> HADOOP-11262-7.patch, HADOOP-11262-8.patch, HADOOP-11262.patch
>
>
> Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[ https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15056258#comment-15056258 ] Sean Mackrory commented on HADOOP-11262: Wanted to add an unofficial +1 - I've done quite a lot of testing on top of this patch and it looks good to me. > Enable YARN to use S3A > --- > > Key: HADOOP-11262 > URL: https://issues.apache.org/jira/browse/HADOOP-11262 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Thomas Demoor >Assignee: Pieter Reuse > Labels: amazon, s3 > Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch, > HADOOP-11262-4.patch, HADOOP-11262-5.patch, HADOOP-11262-6.patch, > HADOOP-11262-7.patch, HADOOP-11262-8.patch, HADOOP-11262.patch > > > Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[
https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998865#comment-14998865
]
Hadoop QA commented on HADOOP-11262:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 7s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 19s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 4s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 10s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 17s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 4s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 57s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 12s
{color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_60.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 26s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 13s
{color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 55s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_60 Failed junit tests | hadoop.metrics2.impl.TestGangliaMetrics |
| JDK v1.7.0_79 Failed junit tests | hadoop.metrics2.impl.TestGangliaMetrics |
\\
\\
|| Subsystem ||
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[
https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14996927#comment-14996927
]
Hadoop QA commented on HADOOP-11262:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 8s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 19s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 5s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 21s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 3s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 25s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 48s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s
{color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_60.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 7s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s
{color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 67m 23s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_60 Failed junit tests | hadoop.fs.shell.TestCopyPreserveFlag |
| JDK v1.7.0_79 Failed junit tests | hadoop.fs.shell.TestCopyPreserveFlag |
| |
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[ https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994094#comment-14994094 ] Lei (Eddy) Xu commented on HADOOP-11262: Thanks for the explanations, [~cnauroth] and [~Thomas Demoor]. Looking forward to hear the updates. > Enable YARN to use S3A > --- > > Key: HADOOP-11262 > URL: https://issues.apache.org/jira/browse/HADOOP-11262 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Thomas Demoor >Assignee: Pieter Reuse > Labels: amazon, s3 > Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch, > HADOOP-11262-4.patch, HADOOP-11262-5.patch, HADOOP-11262-6.patch, > HADOOP-11262-7.patch, HADOOP-11262.patch > > > Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[ https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14989408#comment-14989408 ] Thomas Demoor commented on HADOOP-11262: Thanks for your time reviewing this. [~cnauroth] is correct. In hindsight, probably should have appended "as defaultFS" to the title of this jira. Feel free to adjust it. [~eddyxu], [~PieterReuse] is on holiday this week, we'll address your remarks next week. > Enable YARN to use S3A > --- > > Key: HADOOP-11262 > URL: https://issues.apache.org/jira/browse/HADOOP-11262 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Thomas Demoor >Assignee: Pieter Reuse > Labels: amazon, s3 > Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch, > HADOOP-11262-4.patch, HADOOP-11262-5.patch, HADOOP-11262-6.patch, > HADOOP-11262-7.patch, HADOOP-11262.patch > > > Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[
https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986408#comment-14986408
]
Chris Nauroth commented on HADOOP-11262:
Hi [~eddyxu]. Thanks for joining the code review.
The reason this JIRA mentions enabling YARN to use S3A is that various points
in the YARN codebase rely on the {{FileContext}} API instead of {{FileSystem}}.
An example of this is storing container logs. It is possible to specify an
HDFS URL in {{yarn.nodemanager.log-dirs}}, because HDFS provides a subclass of
{{AbstractFileSystem}}. This is currently not possible with a S3A URL, because
S3A does not provide a subclass of {{AbstractFileSystem}}.
> Enable YARN to use S3A
> ---
>
> Key: HADOOP-11262
> URL: https://issues.apache.org/jira/browse/HADOOP-11262
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Thomas Demoor
>Assignee: Pieter Reuse
> Labels: amazon, s3
> Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch,
> HADOOP-11262-4.patch, HADOOP-11262-5.patch, HADOOP-11262-6.patch,
> HADOOP-11262-7.patch, HADOOP-11262.patch
>
>
> Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[
https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986384#comment-14986384
]
Lei (Eddy) Xu commented on HADOOP-11262:
[~PieterReuse], thanks a lot for working on this. Could you update the code to
obey [Hadoop coding style|https://wiki.apache.org/hadoop/CodeReviewChecklist]
(i.e., 2 space indent and putting {{@override}} below comments.). I think that
we do not need {{@Override}} for {{@Before/@After/@Test}} methods.
{code} try {
// instruct FS to verify checksum through the FileContext:
- fc.setVerifyChecksum(true, path);
out.write(data, 0, data.length);
} finally {
out.close();
+ fc.setVerifyChecksum(true, path);
}
{code}
Could you help me to understand why you move {{setVerifyChecksum}} after
{{close()}}? Wouldn't the checksum not being set in the write op?
{code}
+private String readStringFromFile(Path path){
+FSDataInputStream in = null;
+try {
+in = fs.open(path);
+long bytesLen = fs.getFileStatus(path).getLen();
+byte[] buffer = new byte[(int)bytesLen];
+IOUtils.readFully(in, buffer, 0, buffer.length);
+return new String(buffer);
+} catch (IOException e ){
+throw new RuntimeException("Failed to read from [" + path + "]",
e);
+} finally {
+IOUtils.closeStream(in);
+}
+}
{code}
Could you use {{try-with-resource}} in JDK7 for this? I see you have already
used it in the other functions.
[~ste...@apache.org] and [~cnauroth], could you please provide some color on
this JIRA? You have much more experiences on YARN.
> Enable YARN to use S3A
> ---
>
> Key: HADOOP-11262
> URL: https://issues.apache.org/jira/browse/HADOOP-11262
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Thomas Demoor
>Assignee: Pieter Reuse
> Labels: amazon, s3
> Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch,
> HADOOP-11262-4.patch, HADOOP-11262-5.patch, HADOOP-11262-6.patch,
> HADOOP-11262-7.patch, HADOOP-11262.patch
>
>
> Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[
https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986545#comment-14986545
]
Hadoop QA commented on HADOOP-11262:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 7s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
0s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 22s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 18s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 17s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 22s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 7s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_60.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 12s
{color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_60.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 25s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s
{color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 47s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.1 Server=1.7.1
Image:test-patch-base-hadoop-date2015-11-03 |
| JIRA Patch URL |
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[ https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14907815#comment-14907815 ] Pieter Reuse commented on HADOOP-11262: --- Test failures in hadoop-common unrelated. Patch still applies. > Enable YARN to use S3A > --- > > Key: HADOOP-11262 > URL: https://issues.apache.org/jira/browse/HADOOP-11262 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Thomas Demoor >Assignee: Pieter Reuse > Labels: amazon, s3 > Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch, > HADOOP-11262-4.patch, HADOOP-11262-5.patch, HADOOP-11262-6.patch, > HADOOP-11262-7.patch, HADOOP-11262.patch > > > Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[
https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14703448#comment-14703448
]
Hadoop QA commented on HADOOP-11262:
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 20m 39s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 11 new or modified test files. |
| {color:green}+1{color} | javac | 7m 51s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 52s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | site | 3m 3s | Site still builds. |
| {color:red}-1{color} | checkstyle | 1m 32s | The applied patch generated 1
new checkstyle issues (total was 65, now 62). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 21s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 38s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 41s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | common tests | 22m 50s | Tests failed in
hadoop-common. |
| {color:green}+1{color} | tools/hadoop tests | 0m 14s | Tests passed in
hadoop-aws. |
| | | 71m 8s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.ha.TestZKFailoverController |
| | hadoop.net.TestNetUtils |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12751264/HADOOP-11262-7.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle site |
| git revision | trunk / 2da5aaa |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7510/artifact/patchprocess/diffcheckstylehadoop-aws.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7510/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-aws test log |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7510/artifact/patchprocess/testrun_hadoop-aws.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7510/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7510/console |
This message was automatically generated.
Enable YARN to use S3A
---
Key: HADOOP-11262
URL: https://issues.apache.org/jira/browse/HADOOP-11262
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Reporter: Thomas Demoor
Assignee: Pieter Reuse
Labels: amazon, s3
Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch,
HADOOP-11262-4.patch, HADOOP-11262-5.patch, HADOOP-11262-6.patch,
HADOOP-11262-7.patch, HADOOP-11262.patch
Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[
https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701339#comment-14701339
]
Hadoop QA commented on HADOOP-11262:
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12743479/HADOOP-11262-6.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle site |
| git revision | trunk / 71566e2 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7496/console |
This message was automatically generated.
Enable YARN to use S3A
---
Key: HADOOP-11262
URL: https://issues.apache.org/jira/browse/HADOOP-11262
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Reporter: Thomas Demoor
Assignee: Pieter Reuse
Labels: amazon, s3
Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch,
HADOOP-11262-4.patch, HADOOP-11262-5.patch, HADOOP-11262-6.patch,
HADOOP-11262.patch
Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[ https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14612992#comment-14612992 ] Pieter Reuse commented on HADOOP-11262: --- Whoops, sorry for the double post of this comment. You can ignore one of both. Enable YARN to use S3A --- Key: HADOOP-11262 URL: https://issues.apache.org/jira/browse/HADOOP-11262 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Thomas Demoor Assignee: Pieter Reuse Labels: amazon, s3 Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch, HADOOP-11262-4.patch, HADOOP-11262-5.patch, HADOOP-11262-6.patch, HADOOP-11262.patch Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[ https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578985#comment-14578985 ] Thomas Demoor commented on HADOOP-11262: We (well, [~PieterReuse] actually :p) have implemented all FileContext tests + a test that spins up MiniYARNCluster. Some minor code additions / changes to make things work. Have been running the patch on our test cluster for a week now. So far things look good but want to check some more corner cases before uploading the patch. Enable YARN to use S3A --- Key: HADOOP-11262 URL: https://issues.apache.org/jira/browse/HADOOP-11262 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Thomas Demoor Assignee: Thomas Demoor Labels: amazon, s3 Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch, HADOOP-11262-4.patch, HADOOP-11262-5.patch, HADOOP-11262.patch Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[ https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537792#comment-14537792 ] Thomas Demoor commented on HADOOP-11262: [~jghoman], we're indeed still planning to add these tests. You'll see some progress here from me or [~PieterReuse], who recently joined my team, soon (I want to wrap up some work on HADOOP-9565 first). Enable YARN to use S3A --- Key: HADOOP-11262 URL: https://issues.apache.org/jira/browse/HADOOP-11262 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Thomas Demoor Assignee: Thomas Demoor Labels: amazon, s3 Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch, HADOOP-11262-4.patch, HADOOP-11262-5.patch, HADOOP-11262.patch Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[ https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534875#comment-14534875 ] Jakob Homan commented on HADOOP-11262: -- So, to be clear, [~ste...@apache.org], you're not vetoing the approach, just the patch until more tests are added? [~thodemoor], are you interested in adding more tests? Enable YARN to use S3A --- Key: HADOOP-11262 URL: https://issues.apache.org/jira/browse/HADOOP-11262 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Thomas Demoor Assignee: Thomas Demoor Labels: amazon, s3 Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch, HADOOP-11262-4.patch, HADOOP-11262-5.patch, HADOOP-11262.patch Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[ https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14535377#comment-14535377 ] Steve Loughran commented on HADOOP-11262: - that's right: nothing wrong with the core code, we just need the tests to show they work. Ideally even spinning up a MiniYarnCluster using S3a as the FS Enable YARN to use S3A --- Key: HADOOP-11262 URL: https://issues.apache.org/jira/browse/HADOOP-11262 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Thomas Demoor Assignee: Thomas Demoor Labels: amazon, s3 Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch, HADOOP-11262-4.patch, HADOOP-11262-5.patch, HADOOP-11262.patch Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[ https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14326175#comment-14326175 ] Chris Nauroth commented on HADOOP-11262: bq. Bummer, I simply copied this from the Azure codebase and didn't see any tests there so I assumed this was addressed elsewhere. This is a gap that needs to be addressed in hadoop-azure too, but we can track that in a separate jira. Enable YARN to use S3A --- Key: HADOOP-11262 URL: https://issues.apache.org/jira/browse/HADOOP-11262 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Thomas Demoor Assignee: Thomas Demoor Labels: amazon, s3 Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch, HADOOP-11262-4.patch, HADOOP-11262-5.patch, HADOOP-11262.patch Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[ https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14325799#comment-14325799 ] Thomas Demoor commented on HADOOP-11262: Bummer, I simply copied this from the Azure codebase and didn't see any tests there so I assumed this was addressed elsewhere. Enable YARN to use S3A --- Key: HADOOP-11262 URL: https://issues.apache.org/jira/browse/HADOOP-11262 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Thomas Demoor Assignee: Thomas Demoor Labels: amazon, s3 Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch, HADOOP-11262-4.patch, HADOOP-11262-5.patch, HADOOP-11262.patch Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[
https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14324433#comment-14324433
]
Hadoop QA commented on HADOOP-11262:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12699272/HADOOP-11262-5.patch
against trunk revision f24a567.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5725//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5725//console
This message is automatically generated.
Enable YARN to use S3A
---
Key: HADOOP-11262
URL: https://issues.apache.org/jira/browse/HADOOP-11262
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Reporter: Thomas Demoor
Assignee: Thomas Demoor
Labels: amazon, s3
Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch,
HADOOP-11262-4.patch, HADOOP-11262-5.patch, HADOOP-11262.patch
Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[
https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303555#comment-14303555
]
Hadoop QA commented on HADOOP-11262:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12693262/HADOOP-11262-4.patch
against trunk revision a761bf8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5567//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5567//console
This message is automatically generated.
Enable YARN to use S3A
---
Key: HADOOP-11262
URL: https://issues.apache.org/jira/browse/HADOOP-11262
Project: Hadoop Common
Issue Type: Improvement
Components: fs/s3
Reporter: Thomas Demoor
Assignee: Thomas Demoor
Labels: amazon, s3
Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch,
HADOOP-11262-4.patch, HADOOP-11262.patch
Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[
https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14298918#comment-14298918
]
Chris Nauroth commented on HADOOP-11262:
bq. you shouldn't have to set the s3a.impl file; it should instead go into
core-default.xml
I've been thinking that we really ought to have {{ServiceLoader}} support for
{{AbstractFileSystem}}, so that S3A and others can register themselves in their
jar manifest instead of the configuration files, just like {{FileSystem}}. I
filed HADOOP-11527 for this.
No need to hold up the work happening here waiting for it though. This is just
FYI.
Enable YARN to use S3A
---
Key: HADOOP-11262
URL: https://issues.apache.org/jira/browse/HADOOP-11262
Project: Hadoop Common
Issue Type: Improvement
Components: fs/s3
Reporter: Thomas Demoor
Assignee: Thomas Demoor
Labels: amazon, s3
Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch,
HADOOP-11262-4.patch, HADOOP-11262.patch
Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[
https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280205#comment-14280205
]
Steve Loughran commented on HADOOP-11262:
-
h3. TestS3A
# replace .* import with specific class imports
# you shouldn't have to set the s3a.impl file; it should instead go into
core-default.xml
# {{testS3AStatus()}}: assertTrue/false should include an error message,
ideally with the errant values
# {{testS3ACreateFileInSubDir}} should close FS in a try/finally block. You can
use the Java7 construct to do this now.
Enable YARN to use S3A
---
Key: HADOOP-11262
URL: https://issues.apache.org/jira/browse/HADOOP-11262
Project: Hadoop Common
Issue Type: Improvement
Components: fs/s3
Reporter: Thomas Demoor
Labels: amazon, s3
Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch,
HADOOP-11262.patch
Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
