Steve Loughran created SPARK-51072:
--------------------------------------
Summary: CallerContext to set Hadoop cloud client audit context
Key: SPARK-51072
URL: https://issues.apache.org/jira/browse/SPARK-51072
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.0.0
Reporter: Steve Loughran
CallerContext adds information about the spark task to hadoop IPC context and
then to HDFS, YARN and HBase server logs.
It is also possible to update the cloud storage "audit context".
Storage clients can attach the audit information to requests to be stored in
the service's own logs, where it can be retrieved, parsed and used for analysis.
It is currently supported by the S3A connector, which adds the information to a
synthetic referrer header, which is then stored in the S3 Server logs. (Not
cloudtrail, sadly).
See
https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/auditing.html
CallerContext can be enhanced to attach the generated {{context}} string to the
audit context, so it will propagate to the logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
