[
https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872936#comment-15872936
]
Mingliang Liu commented on HADOOP-13345:
----------------------------------------
Thanks [~fabbri] for prompt reviewing test report!
{quote}
Thats ok.. It does miss ITestS3Guard\{ListConsistency, ToolDynamoDB\}
{quote}
That's a good catch. I just learned these two tests. However,
{{org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency}} fails before/after
merge. Do I need to configure something special?
{code}
cmvn -Dit.test='ITestS3Guard*' -Dtest=none -Dscale -Ds3guard -Ddynamo -q clean
verify
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 14.992 sec <<<
FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
testListStatusWriteBack(org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency)
Time elapsed: 13.552 sec <<< FAILURE!
java.lang.AssertionError: Unexpected number of results from metastore.
Metastore should only know about /XYZ:
DirListingMetadata{path=s3a://mliu-s3guard/test/ListStatusWriteBack,
listMap={s3a://mliu-s3guard/test/ListStatusWriteBack/XYZ=PathMetadata{fileStatus=S3AFileStatus{path=s3a://mliu-s3guard/test/ListStatusWriteBack/XYZ;
isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu;
permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true},
s3a://mliu-s3guard/test/ListStatusWriteBack/123=PathMetadata{fileStatus=S3AFileStatus{path=s3a://mliu-s3guard/test/ListStatusWriteBack/123;
isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu;
permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true}},
isAuthoritative=false}
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testListStatusWriteBack(ITestS3GuardListConsistency.java:127)
{code}
{quote}
Curious, what is our difference in HADOOP-13345 that changes this? Is our
feature branch exception behavior different?
{quote}
We don't really change anything in that part. I guess the reason is that, when
enabling S3Guard, the code path that fails in S3AFileSystem changes for that
test somehow. For example (to be confirmed), the request w/o S3Guard was
calling {{getFileStatus()}} and fails with access denied exception containing
"Forbidden" keyword; while the request w/ S3Guard is able to call
{{getFileStatus()}} and fails later with read operations, which then fails with
access denied exception containing "Access Denied" keyword. So I think relaxing
exception message assertion in test should work just fine.
> S3Guard: Improved Consistency for S3A
> -------------------------------------
>
> Key: HADOOP-13345
> URL: https://issues.apache.org/jira/browse/HADOOP-13345
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Attachments: HADOOP-13345.prototype1.patch, s3c.001.patch,
> S3C-ConsistentListingonS3-Design.pdf, S3GuardImprovedConsistencyforS3A.pdf,
> S3GuardImprovedConsistencyforS3AV2.pdf
>
>
> This issue proposes S3Guard, a new feature of S3A, to provide an option for a
> stronger consistency model than what is currently offered. The solution
> coordinates with a strongly consistent external store to resolve
> inconsistencies caused by the S3 eventual consistency model.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
