[
https://issues.apache.org/jira/browse/HADOOP-14266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967989#comment-15967989
]
Steve Loughran commented on HADOOP-14266:
-----------------------------------------
Let's wait for Aaron & his review.
* Never noticed the code style {{for (; statusIterator.hasNext();)}}. before.
Nice - I'll use it myself
* added a patch with some javadoc tweaks, and made the empty FileStatus entry
returned on all non-s3guard listings a constant.
* testing, s3 ireland.
* got one test failure on the 004 patch, with args {{-Dparallel-tests
-DtestsThreadCount=8 -Ddynamo}}:
{code}
Tests run: 44, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 86.082 sec <<<
FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
testRenameEmptyToDirWithSamePrefixAllowed(org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract)
Time elapsed: 3.221 sec <<< FAILURE!
junit.framework.AssertionFailedError: Rename result expected:<true> but
was:<false>
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:174)
at junit.framework.TestCase.assertEquals(TestCase.java:333)
at
org.apache.hadoop.fs.FileSystemContractBaseTest.rename(FileSystemContractBaseTest.java:567)
at
org.apache.hadoop.fs.FileSystemContractBaseTest.testRenameEmptyToDirWithSamePrefixAllowed(FileSystemContractBaseTest.java:715)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Tests run: 61, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 164.347 sec -
in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations
Results :
Failed tests:
ITestS3AFileSystemContract>FileSystemContractBaseTest.testRenameEmptyToDirWithSamePrefixAllowed:715->FileSystemContractBaseTest.rename:567
Rename result expected:<true> but was:<false>
{code}
This failure implies that attempting to rename an empty dir to itself failed,
when it should be a no-op returning true. Maybe the dir wasn't considered
empty. I could not create this when running
I've managed to get this, with my changes, and after rolling back, with/without
dynamo.
{code}
Tests run: 4, Failures: 0, Errors: 1, Skipped: 3, Time elapsed: 12.364 sec <<<
FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
testConsistentListFiles(org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency)
Time elapsed: 8.944 sec <<< ERROR!
java.io.FileNotFoundException: No such file or directory:
s3a://hwdev-steve-ireland-new/fork-6/test/doTestListFiles-0-0-1-true
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:1852)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:1748)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1710)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerListFiles(S3AFileSystem.java:2435)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.listFiles(S3AFileSystem.java:2418)
at
org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.doTestListFiles(ITestS3GuardListConsistency.java:231)
at
org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testConsistentListFiles(ITestS3GuardListConsistency.java:170)
{code}
> S3Guard: S3AFileSystem::listFiles() to employ MetadataStore
> -----------------------------------------------------------
>
> Key: HADOOP-14266
> URL: https://issues.apache.org/jira/browse/HADOOP-14266
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: HADOOP-13345
> Reporter: Mingliang Liu
> Assignee: Mingliang Liu
> Attachments: HADOOP-14266-HADOOP-13345.000.patch,
> HADOOP-14266-HADOOP-13345.001.patch, HADOOP-14266-HADOOP-13345.002.patch,
> HADOOP-14266-HADOOP-13345.003.patch, HADOOP-14266-HADOOP-13345.003.patch,
> HADOOP-14266-HADOOP-13345.004.patch, HADOOP-14266-HADOOP-13345-005.patch,
> HADOOP-14266-HADOOP-13345.005.patch
>
>
> Similar to [HADOOP-13926], this is to track the effort of employing
> MetadataStore in {{S3AFileSystem::listFiles()}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
