[
https://issues.apache.org/jira/browse/HADOOP-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570193#comment-16570193
]
genericqa commented on HADOOP-15400:
------------------------------------
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
38m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 42s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-15400 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12934469/HADOOP-15400.001.patch
|
| Optional Tests | asflicense mvnsite |
| uname | Linux 5f56b916d4b5 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / bcfc985 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 330 (vs. ulimit of 10000) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14993/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve S3Guard documentation on Authoritative Mode implementation
> ------------------------------------------------------------------
>
> Key: HADOOP-15400
> URL: https://issues.apache.org/jira/browse/HADOOP-15400
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.1
> Reporter: Aaron Fabbri
> Assignee: Gabor Bota
> Priority: Minor
> Attachments: HADOOP-15400.001.patch
>
>
> Part of the design of S3Guard is support for skipping the call to S3
> listObjects and serving directory listings out of the MetadataStore under
> certain circumstances. This feature is called "authoritative" mode. I've
> talked to many people about this feature and it seems to be universally
> confusing.
> I suggest we improve / add a section to the s3guard.md site docs elaborating
> on what Authoritative Mode is.
> It is *not* treating the MetadataStore (e.g. dynamodb) as the source of truth
> in general.
> It *is* the ability to short-circuit S3 list objects and serve listings from
> the MetadataStore in some circumstances:
> For S3A to skip S3's list objects on some *path*, and serve it directly from
> the MetadataStore, the following things must all be true:
> # The MetadataStore implementation persists the bit
> {{DirListingMetadata.isAuthorititative}} set when calling
> {{MetadataStore#put(DirListingMetadata)}}
> # The S3A client is configured to allow metadatastore to be authoritative
> source of a directory listing (fs.s3a.metadatastore.authoritative=true).
> # The MetadataStore has a full listing for *path* stored in it. This only
> happens if the FS client (s3a) explicitly has stored a full directory listing
> with {{DirListingMetadata.isAuthorititative=true}} before the said listing
> request happens.
> Note that #1 only currently happens in LocalMetadataStore. Adding support to
> DynamoDBMetadataStore is covered in HADOOP-14154.
> Also, the multiple uses of the word "authoritative" are confusing. Two
> meanings are used:
> 1. In the FS client configuration fs.s3a.metadatastore.authoritative
> - Behavior of S3A code (not MetadataStore)
> - "S3A is allowed to skip S3.list() when it has full listing from
> MetadataStore"
> 2. MetadataStore
> When storing a dir listing, can set a bit isAuthoritative
> 1 : "full contents of directory"
> 0 : "may not be full listing"
> Note that a MetadataStore *MAY* persist this bit. (not *MUST*).
> We should probably rename the {{DirListingMetadata.isAuthorititative}} to
> {{.fullListing}} or at least put a comment where it is used to clarify its
> meaning.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
