[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15282159#comment-15282159 ] Sangjin Lee commented on HADOOP-9438: - There has been no movement on this for a while. We'll need to move it out of scope for 2.8.0 soon. Let me know if you disagree. > LocalFileContext does not throw an exception on mkdir for already existing > directory > > > Key: HADOOP-9438 > URL: https://issues.apache.org/jira/browse/HADOOP-9438 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.0.3-alpha >Reporter: Robert Joseph Evans >Priority: Critical > Labels: BB2015-05-TBR > Attachments: HADOOP-9438.20130501.1.patch, > HADOOP-9438.20130521.1.patch, HADOOP-9438.patch, HADOOP-9438.patch > > > according to > http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 > should throw a FileAlreadyExistsException if the directory already exists. > I tested this and > {code} > FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); > Path p = new Path("/tmp/bobby.12345"); > FsPermission cachePerms = new FsPermission((short) 0755); > lfc.mkdir(p, cachePerms, false); > lfc.mkdir(p, cachePerms, false); > {code} > never throws an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525897#comment-14525897 ] Hadoop QA commented on HADOOP-9438: --- \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | pre-patch | 14m 33s | Pre-patch trunk compilation is healthy. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | | {color:green}+1{color} | javac | 7m 27s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 9m 35s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 22s | The applied patch does not increase the total number of release audit warnings. | | {color:red}-1{color} | checkstyle | 1m 5s | The applied patch generated 1 new checkstyle issues (total was 38, now 38). | | {color:red}-1{color} | whitespace | 0m 0s | The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. | | {color:green}+1{color} | install | 1m 33s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 1m 39s | The patch does not introduce any new Findbugs (version 2.0.3) warnings. | | {color:green}+1{color} | common tests | 22m 48s | Tests passed in hadoop-common. | | | | 59m 40s | | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12583986/HADOOP-9438.20130521.1.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / a319771 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/6461/artifact/patchprocess/diffcheckstylehadoop-common.txt | | whitespace | https://builds.apache.org/job/PreCommit-HADOOP-Build/6461/artifact/patchprocess/whitespace.txt | | hadoop-common test log | https://builds.apache.org/job/PreCommit-HADOOP-Build/6461/artifact/patchprocess/testrun_hadoop-common.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/6461/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/6461/console | This message was automatically generated. LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.20130501.1.patch, HADOOP-9438.20130521.1.patch, HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525642#comment-14525642 ] Hadoop QA commented on HADOOP-9438: --- (!) The patch artifact directory has been removed! This is a fatal error for test-patch.sh. Aborting. Jenkins (node H4) information at https://builds.apache.org/job/PreCommit-HADOOP-Build/6445/ may provide some hints. LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.20130501.1.patch, HADOOP-9438.20130521.1.patch, HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135405#comment-14135405 ] Steve Loughran commented on HADOOP-9438: ...once we've switched to java 7 we can use the improved java IO methods which will do the test+fail in one go (possibly even atomically) LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.20130501.1.patch, HADOOP-9438.20130521.1.patch, HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135490#comment-14135490 ] Hadoop QA commented on HADOOP-9438: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12583986/HADOOP-9438.20130521.1.patch against trunk revision 7e08c0f. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4737//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4737//console This message is automatically generated. LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.20130501.1.patch, HADOOP-9438.20130521.1.patch, HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984341#comment-13984341 ] Jonathan Eagles commented on HADOOP-9438: - I haven't seen any progress on this issue and can only assume this still exists. I am going to re-target this for a more appropriate line since 2.4.0 is already release and 0.23.11 is now in maintenance mode. LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.20130501.1.patch, HADOOP-9438.20130521.1.patch, HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984445#comment-13984445 ] Hadoop QA commented on HADOOP-9438: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12583986/HADOOP-9438.20130521.1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3868//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3868//console This message is automatically generated. LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.20130501.1.patch, HADOOP-9438.20130521.1.patch, HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13837347#comment-13837347 ] Hadoop QA commented on HADOOP-9438: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12583986/HADOOP-9438.20130521.1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3329//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3329//console This message is automatically generated. LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.20130501.1.patch, HADOOP-9438.20130521.1.patch, HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13704380#comment-13704380 ] Steve Loughran commented on HADOOP-9438: looking at where the code uses this, its a few places in YARN that tend to go {code} if (!fc.exists(path)) { try { fc.mkdirs(path) //some stuff to set permissions up ... } catch (FileAlreadyExistsException fae) { //noop } {code} This catch actually ignores a potentially serious problem -parent path elements not being directories # all FileSystem implementations return true if, at the end of the operation, the directory exists. This isn't in the javadocs of {{FileSystem.mkdirs()}} yet, but I plan to add it. In fact, if you look in {{FSNameSystem.mkdirsInternal()}} the fact that mkdirs() is always expected to succeed is called out {code} // all the users of mkdirs() are used to expect 'true' even if // a new directory is not created. {code} If you look at the implementations of {{FileSystem.mkdirs()}} they will throw either {{FileAlreadyExistsException}} or {{ParentNotDirectoryException}}. Swallowing a {{FileAlreadyExistsException}} can hide a serious problem -which is why MAPREDUCE-5264 is needed. regarding this patch, the interface definition needs to retain the fact that an FAE can be thrown, but that the reason for doing so is the same as for {{ParentNotDirectoryException}}. LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.20130501.1.patch, HADOOP-9438.20130521.1.patch, HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665767#comment-13665767 ] Ivan Mitic commented on HADOOP-9438: Bq. While looking at this, I am starting to suspect that ParentNotDirectoryException doesn't get thrown for a lot of the FileSystem implementations. Hi Steve. This Jira seems pretty well defined at this point so if there is a problem with ParentNotDirectoryException my preference would be to address that separately. Bq. Would we be able to get away with changing the exception that is thrown to this specific error code? As long as things were expecting any IOException this should still work. Not sure I fully understand this part. LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.20130501.1.patch, HADOOP-9438.20130521.1.patch, HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13663970#comment-13663970 ] Rémy SAISSY commented on HADOOP-9438: - I have just created the jira issue: https://issues.apache.org/jira/browse/MAPREDUCE-5264 I will work on it soon. LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.20130501.1.patch, HADOOP-9438.20130521.1.patch, HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664136#comment-13664136 ] Steve Loughran commented on HADOOP-9438: While looking at this, I am starting to suspect that {{ParentNotDirectoryException}} doesn't get thrown for a lot of the {{FileSystem}} implementations. Would we be able to get away with changing the exception that is thrown to this specific error code? As long as things were expecting any {{IOException}} -as the tests do- this should still work. LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.20130501.1.patch, HADOOP-9438.20130521.1.patch, HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664881#comment-13664881 ] Ivan Mitic commented on HADOOP-9438: bq. I have just created the jira issue: https://issues.apache.org/jira/browse/MAPREDUCE-5264. I will work on it soon. Thanks Remy, much appreciated! LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.20130501.1.patch, HADOOP-9438.20130521.1.patch, HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662849#comment-13662849 ] Hadoop QA commented on HADOOP-9438: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12583986/HADOOP-9438.20130521.1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2553//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2553//console This message is automatically generated. LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.20130501.1.patch, HADOOP-9438.20130521.1.patch, HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13663199#comment-13663199 ] Robert Joseph Evans commented on HADOOP-9438: - I think the patch looks fine and I am +1 for this, but I really would like someone who is much more on the HDFS side to also take a look before checking it in. Especially because this is technically an incompatible change. LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.20130501.1.patch, HADOOP-9438.20130521.1.patch, HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13663686#comment-13663686 ] Ivan Mitic commented on HADOOP-9438: Patch looks good to me as well, +1. I will also defer to someone from HDFS side to make the final call. Btw, I noticed a couple of places in the mapreduce codebase where we coded with assumption that FileSystem#mkdirs or FileContext#mkdir throw FileAlreadyExistsException if the directory already exist. From a quick glance, I saw this pattern in the following files: - YarnChild.java, JobHistoryEverntHandler.java and HistoryFileManager.java Would be good to re-review and patch this if needed. [~rémy] do you maybe want to file a Jira and/or work on this? LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.20130501.1.patch, HADOOP-9438.20130521.1.patch, HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13649120#comment-13649120 ] Ivan Mitic commented on HADOOP-9438: I agree with Robert, changing the javadoc and the interface is the preferable fix for the reasons mentioned. The behavior of LocalFs and Hdfs is consistent in this scenario. Would be good to do a quick scan thru the Hadoop codebase for FileAlreadyExistsExceptionand and see if there are some other changes that need to happen. Not sure if there is something else to worry about? LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.20130501.1.patch, HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13648259#comment-13648259 ] Rémy SAISSY commented on HADOOP-9438: - It breaks a lot of tests. Now you mention it, indeed the FileAlreadyExistsException is only on the FileContext.mkdir javadoc. The FileSystem contract and its RawLocalFileSystem implementation does not throw such an exception. However, the AbstractFileSystem.mkdirs method signature throws a FileAlreadyExistsException. So maybe it should be implemented only in FileContext.mkdir(). Does something like this would be ok in FileContext.java.mkdirs()? (and adding exists() to the AbstractFileSystem contract). {code} new FSLinkResolverVoid() { @Override public Void next(final AbstractFileSystem fs, final Path p) throws IOException, UnresolvedLinkException { if (fs.exists(p)) { throw new FileAlreadyExistsException(p + already exists.); } fs.mkdir(p, absFerms, createParent); return null; } }.resolve(this, absDir); {code} I don't really understand what is the difference between AbstracFileSystem and FileSystem? I didn't notice it at first but it looks like they implement almost the same thing. Isn't it overlapping and shouldn't it be refactored? LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.20130501.1.patch, HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13648443#comment-13648443 ] Robert Joseph Evans commented on HADOOP-9438: - Like I said initially I am fine with having the javadocs and interface for FileContext.mkdir changed. The problem isn't that FileContext does not throw the exception the problem is that it is inconsistent with the documented interface. This resulted in incorrect code being written in YARN. The above code is not going to solve the problem because there is a race. If fs.exists returns false and then the file is created the mkdir will become a noop and the interface still is not followed. Changing the FileSystem definition to throw an exception on a mkdir is not acceptable either. This will not just break tests it will break lots of downstream customers. The trick with changing the FileContext definition is that we have to be sure that it is in line with the other implementations as well. If all of the FileContext implementations are wrappers around FileSystem implementations then it should not be a problem to change this. LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.20130501.1.patch, HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13647919#comment-13647919 ] Ivan Mitic commented on HADOOP-9438: I run into this Jira and thought it might be worth to comment. Per the FileSystem contract, mkdirs should succeed if the directory already exist, right? Existing FileSystemContract tests validate this functionality. Is this something that should be handled by the FileContext, or possibly a documentation bug? LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.20130501.1.patch, HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646985#comment-13646985 ] Omkar Vinit Joshi commented on HADOOP-9438: --- Yes you are right... when I looked at the code I was more worried about the existing functionalities which are making use of that method directly / indirectly via FileContext. [~arpitgupta] is the RawLocalFileSystem.mkdirs(Path) method getting used anywhere internally in the following context? if directory is present silently return? if yes then probably we either need another method which will be called from FileContext as otherwise it will break existing functionalities. LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.20130501.1.patch, HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646991#comment-13646991 ] Arpit Gupta commented on HADOOP-9438: - [~ojoshi] i think you wanted to tag [~arpitagarwal] LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.20130501.1.patch, HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13647011#comment-13647011 ] Omkar Vinit Joshi commented on HADOOP-9438: --- yes :) ..sorry for this.. LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.20130501.1.patch, HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13647013#comment-13647013 ] Hadoop QA commented on HADOOP-9438: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12581422/HADOOP-9438.20130501.1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 12 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.fs.viewfs.TestViewFsTrash org.apache.hadoop.fs.TestFcLocalFsUtil org.apache.hadoop.fs.TestTrash {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2505//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2505//console This message is automatically generated. LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.20130501.1.patch, HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642806#comment-13642806 ] Hadoop QA commented on HADOOP-9438: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12580688/HADOOP-9438.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 6 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.fs.viewfs.TestFcPermissionsLocalFs org.apache.hadoop.fs.viewfs.TestFcCreateMkdirLocalFs org.apache.hadoop.fs.TestFcLocalFsUtil {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2478//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2478//console This message is automatically generated. LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643075#comment-13643075 ] Omkar Vinit Joshi commented on HADOOP-9438: --- [~rémy] I see few problems with current implementation. * Why is the check added to FileSystem?? This is good if we check it upfront but better it will be to delegate the call to RawLocalFileSystem or any other file system and check the return or exception type. At present there exists a race condition. This code will work if we receive mkdir requests one after the other but will not work if they arrive at the same time. Did you take a look at the below method? I suspect that fixing this will solve the whole problem. What do you think? This method will either create the directory or will silently return if the directory is already present. {code:title=RawLocalFileSystem.java|borderStyle=solid} public boolean mkdirs(Path f) throws IOException { return (parent == null || mkdirs(parent)) (p2f.mkdir() || p2f.isDirectory()); } {code} * All the Test code classes are created with different name formats. for unit test Please create classes Test and methods as test LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643090#comment-13643090 ] Omkar Vinit Joshi commented on HADOOP-9438: --- also a small suggestion..( which I got in the beginning.. :) )... please create patch name as issue-number.MMDD.NUM.patch ex. HADOOP-9438.20130426.patch, HADOOP-9438.20130426.1.patch This will definitely help reviewers review the patch. LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643238#comment-13643238 ] Rémy SAISSY commented on HADOOP-9438: - Thanks for the suggestion, I will name the patches this way :). I took a look at the method you mentioned but I was not sure that it was the deepest place in the call stack. The debugger led me to primitiveMkdir() and I stopped there. Indeed it looks like it is too upfront so I will move this code closer to the actual creation of the directory/file. The patch required me to change some code in many unit tests and I have noticed an issue with symlinks (maybe it should be another JIRA issue). The java.io.File.exist() method returns False when checking a broken symlink. Even the org.apache.commons.io.FileUtils.isSymlink() method doesn't work in this case. LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.patch, HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639726#comment-13639726 ] Omkar Vinit Joshi commented on HADOOP-9438: --- [~rémy] I see that you have reformatted lot of code (not related to this jira). can you please update the patch with git diff showing only the changes related to your changes? LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639775#comment-13639775 ] Hadoop QA commented on HADOOP-9438: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12580155/HADOOP-9438.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.fs.TestLocalFSFileContextMainOperations org.apache.hadoop.fs.viewfs.TestFcPermissionsLocalFs org.apache.hadoop.fs.TestLocal_S3FileContextURI org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs org.apache.hadoop.fs.viewfs.TestFcCreateMkdirLocalFs org.apache.hadoop.fs.TestS3_LocalFileContextURI org.apache.hadoop.fs.TestFcLocalFsUtil org.apache.hadoop.fs.TestFileContextResolveAfs {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2468//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2468//console This message is automatically generated. LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Robert Joseph Evans Priority: Critical Attachments: HADOOP-9438.patch according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13616330#comment-13616330 ] Robert Joseph Evans commented on HADOOP-9438: - Thanks Omkar, This and HDFS-4619 look like they may be dupes. LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Reporter: Robert Joseph Evans Priority: Critical according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13614441#comment-13614441 ] omkar vinit joshi commented on HADOOP-9438: --- this is the related jira [HDFS-4619|https://issues.apache.org/jira/browse/HDFS-4619] LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Reporter: Robert Joseph Evans Priority: Critical according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9438) LocalFileContext does not throw an exception on mkdir for already existing directory
[ https://issues.apache.org/jira/browse/HADOOP-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13614490#comment-13614490 ] Arpit Agarwal commented on HADOOP-9438: --- +1 for the idea. One round trip is better than two. However it would be good to ensure that implementations of AbstractFileSystem.mkdir behave consistently. I cannot find an analog to FileSystemContractBaseTest for AbstractFileSystem. Interestingly FileSystem.mkdirs does not fail for pre-existing directories. LocalFileContext does not throw an exception on mkdir for already existing directory Key: HADOOP-9438 URL: https://issues.apache.org/jira/browse/HADOOP-9438 Project: Hadoop Common Issue Type: Bug Reporter: Robert Joseph Evans Priority: Critical according to http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#mkdir%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.permission.FsPermission,%20boolean%29 should throw a FileAlreadyExistsException if the directory already exists. I tested this and {code} FileContext lfc = FileContext.getLocalFSFileContext(new Configuration()); Path p = new Path(/tmp/bobby.12345); FsPermission cachePerms = new FsPermission((short) 0755); lfc.mkdir(p, cachePerms, false); lfc.mkdir(p, cachePerms, false); {code} never throws an exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira