[
https://issues.apache.org/jira/browse/FLINK-16809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17067665#comment-17067665
]
zhuqi commented on FLINK-16809:
-------------------------------
Hi [~sewen]
Thanks for your reply, i linked some related issues.
Such as first step we can pass jobId to Task.
Invoke Hadoop APIs.
Add a new function invokes APIs of org.apache.hadoop.ipc.CallerContext to set
up flink caller contexts, which will be written into hdfs-audit.log and Yarn RM
audit log.
For HDFS: Flink sets up its caller context by
invokingorg.apache.hadoop.ipc.CallerContext in Task and Yarn Client and
ApplicationMaster.
For Yarn: Flink sets up its caller context by invoking
org.apache.hadoop.ipc.CallerContext in Yarn Client.
Some like in spark :
[https://github.com/apache/spark/pull/14659]
> Add Caller Context in Flink
> ---------------------------
>
> Key: FLINK-16809
> URL: https://issues.apache.org/jira/browse/FLINK-16809
> Project: Flink
> Issue Type: Improvement
> Reporter: zhuqi
> Priority: Major
>
> Now the Spark and hive have the Call Context to meet the HDFS Job Audit
> requirement.
> I think the flink also should to add it, and in our cluster the flink job may
> have big pressure to the HDFS, it's will be helpful to find the root job.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
