Steve Loughran created HADOOP-16697:

             Summary: audit/tune s3a authoritative flag in s3guard DDB Table
                 Key: HADOOP-16697
             Project: Hadoop Common
          Issue Type: Sub-task
          Components: fs/s3
    Affects Versions: 3.3.0
            Reporter: Steve Loughran
            Assignee: Steve Loughran

S3A auth mode can cause confusion in deployments, because people expect there 
never to be any HTTP requests to S3 in a path marked as authoritative.

This is *not* the case when S3Guard doesn't have an entry for the path in the 
table. Which is the state it is in when the directory was populated using 
different tools (e.g AWS s3 command).


1. HADOOP-16684 to give more diagnostics about the bucket

2. add an audit command to take a path and verify that it is marked in dynamoDB 
as authoritative *all the way down*

This command is designed to be executed from the commandline and will return 
different error codes based on different situations

* path isn't guarded
* path is not authoritative in s3a settings (dir, path)
* path not known in table: use the 404/44 response
* path contains 1+ dir entry which is non-auth

3. Use this audit after some of the bulk rename, delete, import, commit (soon: 
upload, copy) operations to verify that's where appropriate, we do update the 
directories. Particularly for incremental rename() where I have long suspected 
we may have to do more there.

4. Review documentation and make it clear what is needed (import) after 
uploading/Generating Data through other tools.

I'm going to pull in the open JIRAs on this topic as they are all related

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to