[jira] [Commented] (HADOOP-13188) S3A file-create should throw error rather than overwrite directories
[ https://issues.apache.org/jira/browse/HADOOP-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392095#comment-15392095 ] Hudson commented on HADOOP-13188: - SUCCESS: Integrated in Hadoop-trunk-Commit #10144 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10144/]) HADOOP-13188 S3A file-create should throw error rather than overwrite (stevel: rev 86ae218893d018638e937c2528c8e84336254da7) * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/TestS3AContractCreate.java > S3A file-create should throw error rather than overwrite directories > > > Key: HADOOP-13188 > URL: https://issues.apache.org/jira/browse/HADOOP-13188 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Raymie Stata >Assignee: Steve Loughran >Priority: Minor > Fix For: 2.8.0 > > Attachments: HADOOP-13188-branch-2-001.patch > > > S3A.create(Path,FsPermission,boolean,int,short,long,Progressable) is not > checking to see if it's being asked to overwrite a directory. It could > easily do so, and should throw an error in this case. > There is a test-case for this in AbstractFSContractTestBase, but it's being > skipped because S3A is a blobstore. However, both the Azure and Swift file > systems make this test, and the new S3 one should as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13188) S3A file-create should throw error rather than overwrite directories
[ https://issues.apache.org/jira/browse/HADOOP-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336655#comment-15336655 ] Steve Loughran commented on HADOOP-13188: - thanks, I 'll commit it over the weekend. if you get S3a.toString() on your FS instance, you'll get the full io stats printed out.Be interesting to see the changes. I don't think you should be seeing any perf diff. There's a call to exists(path), which is getFileStatus in a try/catch block. Same overhead, simply without check for the path referring to a directory if it is there > S3A file-create should throw error rather than overwrite directories > > > Key: HADOOP-13188 > URL: https://issues.apache.org/jira/browse/HADOOP-13188 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Raymie Stata >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-13188-branch-2-001.patch > > > S3A.create(Path,FsPermission,boolean,int,short,long,Progressable) is not > checking to see if it's being asked to overwrite a directory. It could > easily do so, and should throw an error in this case. > There is a test-case for this in AbstractFSContractTestBase, but it's being > skipped because S3A is a blobstore. However, both the Azure and Swift file > systems make this test, and the new S3 one should as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13188) S3A file-create should throw error rather than overwrite directories
[
https://issues.apache.org/jira/browse/HADOOP-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336614#comment-15336614
]
Ravi Prakash commented on HADOOP-13188:
---
Looks good to me. +1. FWIW, I was concerned about the performance penalty.
So I uploaded 16 files (using {{hadoop fs -put
s3a://:@bucket/path}}). With the patch it took
{code}
real2m16.564s
user0m11.571s
sys 0m0.582s
{code}
Without the patch:
{code}
real2m5.481s
user0m10.811s
sys 0m0.472s{code}
So its not that bad a degradation.. Still pretty bad performance though, but
that is clearly another JIRA
> S3A file-create should throw error rather than overwrite directories
>
>
> Key: HADOOP-13188
> URL: https://issues.apache.org/jira/browse/HADOOP-13188
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.2
>Reporter: Raymie Stata
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13188-branch-2-001.patch
>
>
> S3A.create(Path,FsPermission,boolean,int,short,long,Progressable) is not
> checking to see if it's being asked to overwrite a directory. It could
> easily do so, and should throw an error in this case.
> There is a test-case for this in AbstractFSContractTestBase, but it's being
> skipped because S3A is a blobstore. However, both the Azure and Swift file
> systems make this test, and the new S3 one should as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13188) S3A file-create should throw error rather than overwrite directories
[ https://issues.apache.org/jira/browse/HADOOP-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15307590#comment-15307590 ] Steve Loughran commented on HADOOP-13188: - HADOOP-13221 relates to this; a fix from that needs to merge in a check for the dest path being a directory too > S3A file-create should throw error rather than overwrite directories > > > Key: HADOOP-13188 > URL: https://issues.apache.org/jira/browse/HADOOP-13188 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Raymie Stata >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-13188-branch-2-001.patch > > > S3A.create(Path,FsPermission,boolean,int,short,long,Progressable) is not > checking to see if it's being asked to overwrite a directory. It could > easily do so, and should throw an error in this case. > There is a test-case for this in AbstractFSContractTestBase, but it's being > skipped because S3A is a blobstore. However, both the Azure and Swift file > systems make this test, and the new S3 one should as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13188) S3A file-create should throw error rather than overwrite directories
[
https://issues.apache.org/jira/browse/HADOOP-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298273#comment-15298273
]
Hadoop QA commented on HADOOP-13188:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
36s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 11s
{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s
{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 18s
{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s
{color} | {color:red} hadoop-tools/hadoop-aws: The patch generated 1 new + 14
unchanged - 0 fixed = 15 total (was 14) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 10s
{color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_91.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 10s
{color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_101.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 40s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:babe025 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12805883/HADOOP-13188-branch-2-001.patch
|
| JIRA Issue | HADOOP-13188 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b2968ed95eaf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
|
