[
https://issues.apache.org/jira/browse/FLINK-25029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17448457#comment-17448457
]
David Morávek commented on FLINK-25029:
---------------------------------------
I'd say this is mostly related to filesystems (namely to the hadoop
filesystem). If I understand it correctly, it should be enough to call
`CallerContext#setCurrent` before initiating any calls that interact with HDFS.
Whether it's also a coordination issue depends on whether filesystems have
enough information to create a "descriptive" context for the call. (eg. in
context hierarchy, "flink -> <jobId> -> <taskId> -> ???")
[1]
https://github.com/apache/hadoop/blob/rel/release-2.8.5/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallerContext.java#L147
> Hadoop Caller Context Setting In Flink
> --------------------------------------
>
> Key: FLINK-25029
> URL: https://issues.apache.org/jira/browse/FLINK-25029
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
> Reporter: 刘方奇
> Priority: Major
>
> For a given HDFS operation (e.g. delete file), it's very helpful to track
> which upper level job issues it. The upper level callers may be specific
> Oozie tasks, MR jobs, and hive queries. One scenario is that the namenode
> (NN) is abused/spammed, the operator may want to know immediately which MR
> job should be blamed so that she can kill it. To this end, the caller context
> contains at least the application-dependent "tracking id".
> The above is the main effect of the Caller Context. HDFS Client set Caller
> Context, then name node get it in audit log to do some work.
> Now the Spark and hive have the Caller Context to meet the HDFS Job Audit
> requirement.
> In my company, flink jobs often cause some problems for HDFS, so we did it
> for preventing some cases.
> If the feature is general enough. Should we support it, then I can submit a
> PR for this.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
