[
https://issues.apache.org/jira/browse/HADOOP-14107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880310#comment-15880310
]
Steve Loughran commented on HADOOP-14107:
-----------------------------------------
* Is the test doing the write thing? It's asserting that the noWriteBack
metastore should only see the entry it created, *and not that created by
yesWriteback*
Except shouldn't it be that because yesWriteback is writing back the entry,
listing operations which do still go to s3 may pick up the created entry? What
Mingliang may be seeing on his runs is that the S3 LIST command is up to date
with the written back entry, so return both. Other people (including myself)
have our tests working because we are doing the LIST before the inconsistent s3
listing has grabbed the entry
Looking at the test, I've some recommendations to aid diagnostics
# we add a small delay between write and probe, so that s3 will list stuff if
it has it.
# the asserts should really be assertEquals, though that isn't the cause of
this problem
# the names of the paths should be writtenBack and notWrittenBack not XYZ and
123. That will make what's going on clearer
# assert text to indicate whether its writeback/not writeback
> ITestS3GuardListConsistency fails intermittently
> ------------------------------------------------
>
> Key: HADOOP-14107
> URL: https://issues.apache.org/jira/browse/HADOOP-14107
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: HADOOP-13345
> Reporter: Mingliang Liu
>
> {code}
> mvn -Dit.test='ITestS3GuardListConsistency' -Dtest=none -Dscale -Ds3guard
> -Ddynamo -q clean verify
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.544 sec <<<
> FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
> testListStatusWriteBack(org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency)
> Time elapsed: 3.147 sec <<< FAILURE!
> java.lang.AssertionError: Unexpected number of results from metastore.
> Metastore should only know about /XYZ:
> DirListingMetadata{path=s3a://mliu-s3guard/test/ListStatusWriteBack,
> listMap={s3a://mliu-s3guard/test/ListStatusWriteBack/XYZ=PathMetadata{fileStatus=S3AFileStatus{path=s3a://mliu-s3guard/test/ListStatusWriteBack/XYZ;
> isDirectory=true; modification_time=0; access_time=0; owner=mliu;
> group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true},
> s3a://mliu-s3guard/test/ListStatusWriteBack/123=PathMetadata{fileStatus=S3AFileStatus{path=s3a://mliu-s3guard/test/ListStatusWriteBack/123;
> isDirectory=true; modification_time=0; access_time=0; owner=mliu;
> group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true}},
> isAuthoritative=false}
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testListStatusWriteBack(ITestS3GuardListConsistency.java:127)
> {code}
> See discussion on the parent JIRA [HADOOP-13345].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
