Steve Loughran created HADOOP-16697:
---------------------------------------
Summary: audit/tune s3a authoritative flag in s3guard DDB Table
Key: HADOOP-16697
URL: https://issues.apache.org/jira/browse/HADOOP-16697
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.3.0
Reporter: Steve Loughran
Assignee: Steve Loughran
S3A auth mode can cause confusion in deployments, because people expect there
never to be any HTTP requests to S3 in a path marked as authoritative.
This is *not* the case when S3Guard doesn't have an entry for the path in the
table. Which is the state it is in when the directory was populated using
different tools (e.g AWS s3 command).
Proposed
1. HADOOP-16684 to give more diagnostics about the bucket
2. add an audit command to take a path and verify that it is marked in dynamoDB
as authoritative *all the way down*
This command is designed to be executed from the commandline and will return
different error codes based on different situations
* path isn't guarded
* path is not authoritative in s3a settings (dir, path)
* path not known in table: use the 404/44 response
* path contains 1+ dir entry which is non-auth
3. Use this audit after some of the bulk rename, delete, import, commit (soon:
upload, copy) operations to verify that's where appropriate, we do update the
directories. Particularly for incremental rename() where I have long suspected
we may have to do more there.
4. Review documentation and make it clear what is needed (import) after
uploading/Generating Data through other tools.
I'm going to pull in the open JIRAs on this topic as they are all related
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
