[
https://issues.apache.org/jira/browse/HADOOP-13998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15849666#comment-15849666
]
Aaron Fabbri edited comment on HADOOP-13998 at 2/2/17 8:56 AM:
---------------------------------------------------------------
All of the dependencies for this have patches available for review. I would
like to start on empty directory handling improvements, but would prefer to
merge the feature branch to trunk first to avoid having to maintain more
S3AFileSystem diffs.
*I'm proposing we merge* HADOOP-13345 to trunk as soon as we get the dependent
JIRAs linked here committed. I'll provide a summary of where we are at below.
I look forward to feedback from [[email protected]], [~cnauroth], [~eddyxu],
[~mackrorysd], and the rest of the community.
The main feature we want for the initial version is listing consistency, and
we've accomplished that.
For testing, we have completed (off the top of my head):
- List consistency tests with failure injection. (HADOOP-13793) This
integration test forces a delay in visibility of certain files by wrapping the
AWS S3 client. It asserts listing is consistent. The test fails without
S3Guard, and succeeds with it.
- All existing S3 integration tests with and without S3Guard. The filesystem
contract tests have been invaluable here. (HADOOP-13589 makes these very easy
to run).
- MetadataStore contract tests that ensure that the API semantics of the
DynamoDB and in-memory reference implementations are correct.
- MetadataStore scale tests that can be used to force DynamoDB service
throttling and ensure we are robust to that.
- Unit tests for different parts of the S3Guard logic.
In addition to this upstream testing, my colleagues have run a couple of our
in-house test harnesses against S3Guard. This includes Hive, Spark, and a
number of other components. All the testing is looking great so far.
Edit: Here is a [link to current s3guard
documentation|https://github.com/apache/hadoop/blob/HADOOP-13345/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md]
was (Author: fabbri):
All of the dependencies for this have patches available for review. I would
like to start on empty directory handling improvements, but would prefer to
merge the feature branch to trunk first to avoid having to maintain more
S3AFileSystem diffs.
*I'm proposing we merge* HADOOP-13345 to trunk as soon as we get the dependent
JIRAs linked here committed. I'll provide a summary of where we are at below.
I look forward to feedback from [[email protected]], [~cnauroth], [~eddyxu],
[~mackrorysd], and the rest of the community.
The main feature we want for the initial version is listing consistency, and
we've accomplished that.
For testing, we have completed (off the top of my head):
- List consistency tests with failure injection. (HADOOP-13793) This
integration test forces a delay in visibility of certain files by wrapping the
AWS S3 client. It asserts listing is consistent. The test fails without
S3Guard, and succeeds with it.
- All existing S3 integration tests with and without S3Guard. The filesystem
contract tests have been invaluable here. (HADOOP-13589 makes these very easy
to run).
- MetadataStore contract tests that ensure that the API semantics of the
DynamoDB and in-memory reference implementations are correct.
- MetadataStore scale tests that can be used to force DynamoDB service
throttling and ensure we are robust to that.
- Unit tests for different parts of the S3Guard logic.
In addition to this upstream testing, my colleagues have run a couple of our
in-house test harnesses against S3Guard. This includes Hive, Spark, and a
number of other components. All the testing is looking great so far.
> initial s3guard preview
> -----------------------
>
> Key: HADOOP-13998
> URL: https://issues.apache.org/jira/browse/HADOOP-13998
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Steve Loughran
>
> JIRA to link in all the things we think are needed for a preview/merge into
> trunk
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
