Steve Loughran created HADOOP-18659:
---------------------------------------
Summary: ABFS to add Auditing support and attach http referrer to
requests
Key: HADOOP-18659
URL: https://issues.apache.org/jira/browse/HADOOP-18659
Project: Hadoop Common
Issue Type: New Feature
Components: fs/azure
Affects Versions: 3.3.5
Reporter: Steve Loughran
the s3a auditing feature of HADOOP-17511 is wonderful in production, as we can
* scan for all IO done by a single user, job, operation
* get a comprehensive view of what was done to the store, in the order it saw
it, which doesn't always match the order the client logged it, especially
logged outcomes as that only happens when the threads get scheduled.
the abfs TracingContext effectively does most of this. All that is needed is to
integrate with the hadoop-common audit interfaces/apis
* TracingContext to pick up global/thread-local audit contexts in construction
and so extract information (tool class; principal; spark job id. etc)
The manifest committer already sets task id on the active thread whenever
entered; this persists through all the IO done by the spark worker.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
