[
https://issues.apache.org/jira/browse/HADOOP-12169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15222419#comment-15222419
]
Steve Loughran commented on HADOOP-12169:
-----------------------------------------
I've done a run of this, got 4 s3a errors. I'm not (yet) sure whether they are
related or not
{code}
testListEmptyRootDirectory(org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir)
Time elapsed: 4.886 sec <<< ERROR!
java.io.IOException: Root directory operation rejected: s3a://tests3neu/
at
org.apache.hadoop.fs.contract.ContractTestUtils.rejectRootOperation(ContractTestUtils.java:375)
at
org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:558)
at
org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:541)
at
org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testListEmptyRootDirectory(AbstractContractRootDirectoryTest.java:132)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 93.393 sec -
in org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractCreate
Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 29.563 sec - in
org.apache.hadoop.fs.contract.s3n.TestS3NContractCreate
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractDelete
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.541 sec - in
org.apache.hadoop.fs.contract.s3n.TestS3NContractDelete
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractMkdir
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.355 sec - in
org.apache.hadoop.fs.contract.s3n.TestS3NContractMkdir
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractOpen
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.969 sec - in
org.apache.hadoop.fs.contract.s3n.TestS3NContractOpen
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractRename
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.031 sec - in
org.apache.hadoop.fs.contract.s3n.TestS3NContractRename
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractRootDir
Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 39.601 sec <<<
FAILURE! - in org.apache.hadoop.fs.contract.s3n.TestS3NContractRootDir
testListEmptyRootDirectory(org.apache.hadoop.fs.contract.s3n.TestS3NContractRootDir)
Time elapsed: 2.05 sec <<< ERROR!
java.lang.IllegalArgumentException: Can not create a Path from an empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:125)
at org.apache.hadoop.fs.Path.<init>(Path.java:133)
at org.apache.hadoop.fs.Path.<init>(Path.java:92)
at
org.apache.hadoop.fs.s3native.NativeS3FileSystem.listStatus(NativeS3FileSystem.java:560)
at
org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testListEmptyRootDirectory(AbstractContractRootDirectoryTest.java:130)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractSeek
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 62.854 sec -
in org.apache.hadoop.fs.contract.s3n.TestS3NContractSeek
Running org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract
Tests run: 31, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.267 sec - in
org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract
Running org.apache.hadoop.fs.s3.TestINode
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.054 sec - in
org.apache.hadoop.fs.s3.TestINode
Running org.apache.hadoop.fs.s3.TestS3Credentials
Tests run: 6, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 1.126 sec <<<
FAILURE! - in org.apache.hadoop.fs.s3.TestS3Credentials
noSecretShouldThrow(org.apache.hadoop.fs.s3.TestS3Credentials) Time elapsed:
0.06 sec <<< FAILURE!
java.lang.AssertionError: Expected exception: java.lang.IllegalArgumentException
at
org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:32)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
noAccessIdShouldThrow(org.apache.hadoop.fs.s3.TestS3Credentials) Time elapsed:
0.037 sec <<< FAILURE!
java.lang.AssertionError: Expected exception: java.lang.IllegalArgumentException
at
org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:32)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Tests in error:
TestS3ContractRootDir>AbstractContractRootDirectoryTest.testRmEmptyRootDirNonRecursive:67
» FileNotFound
TestS3ContractRootDir>AbstractContractRootDirectoryTest.testRmRootRecursive:101
» FileNotFound
TestS3AContractRootDir>AbstractContractRootDirectoryTest.testListEmptyRootDirectory:132
» IO
TestS3NContractRootDir>AbstractContractRootDirectoryTest.testListEmptyRootDirectory:130
» IllegalArgument
Tests run: 371, Failures: 2, Errors: 4, Skipped: 11
{code}
> ListStatus on empty dir in S3A lists itself instead of returning an empty list
> ------------------------------------------------------------------------------
>
> Key: HADOOP-12169
> URL: https://issues.apache.org/jira/browse/HADOOP-12169
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.6.0, 2.7.0, 2.7.1
> Reporter: Pieter Reuse
> Assignee: Pieter Reuse
> Attachments: HADOOP-12169-001.patch, HADOOP-12169-002.patch
>
>
> Upon testing the patch for HADOOP-11918, I stumbled upon a weird behaviour
> this introduces to the S3AFileSystem-class. Calling ListStatus() on an empty
> bucket returns an empty list, while doing the same on an empty directory,
> returns an array of length 1 containing only this directory itself.
> The bugfix is quite simple. In the line of code {code}...if
> (keyPath.equals(f)...{code} (S3AFileSystem:758), keyPath is qualified wrt.
> the fs and f is not. Therefore, this returns false while it shouldn't. The
> bugfix to make f qualified in this line of code.
> More formally: accoring to the formal definition of [The Hadoop FileSystem
> API
> Definition|https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/filesystem/],
> more specifically FileSystem.listStatus, only child elements of a directory
> should be returned upon a listStatus()-call.
> In detail:
> {code}
> elif isDir(FS, p): result [getFileStatus(c) for c in children(FS, p) where
> f(c) == True]
> {code}
> and
> {code}
> def children(FS, p) = {q for q in paths(FS) where parent(q) == p}
> {code}
> Which translates to the result of listStatus on an empty directory being an
> empty list. This is the same behaviour as ls has in Unix, which is what
> someone would expect from a FileSystem.
> Note: it seemed appropriate to add the test of this patch to the same file as
> the test for HADOOP-11918, but as a result, one of the two will have to be
> rebased wrt. the other before being applied to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
