Github user tgravescs commented on a diff in the pull request:
    --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
    @@ -2420,29 +2420,54 @@ private[spark] object Utils extends Logging {
    + * An utility class used to set up Spark caller contexts to HDFS and Yarn. 
The `context` will be
    + * constructed by parameters passed in.
    + * When Spark applications run on Yarn and HDFS, its caller contexts will 
be written into Yarn RM
    + * audit log and hdfs-audit.log. That can help users to better diagnose 
and understand how
    + * specific applications impacting parts of the Hadoop system and 
potential problems they may be
    + * creating (e.g. overloading NN). As HDFS mentioned in HDFS-9184, for a 
given HDFS operation, it's
    + * very helpful to track which upper level job issues it.
    + *
    + * @param from who sets up the caller context (TASK, CLIENT, 
    + *
    + * The parameters below are optional:
    + * @param appID id of the app this task belongs to
    + * @param appAttemptID attempt id of the app this task belongs to
    + * @param jobID id of the job this task belongs to
    + * @param stageID id of the stage this task belongs to
    + * @param stageAttemptId attempt id of the stage this task belongs to
    + * @param taskId task id
    + * @param taskAttemptNumber task attempt id
    + * @since 2.0.1
    + */
     private[spark] class CallerContext(
    -   appName: Option[String] = None,
    -   appID: Option[String] = None,
    -   appAttemptID: Option[String] = None,
    -   jobID: Option[Int] = None,
    -   stageID: Option[Int] = None,
    +   from: String,
    +   appId: Option[String] = None,
    +   appAttemptId: Option[String] = None,
    +   jobId: Option[Int] = None,
    +   stageId: Option[Int] = None,
        stageAttemptId: Option[Int] = None,
        taskId: Option[Long] = None,
        taskAttemptNumber: Option[Int] = None) extends Logging {
    -   val AppName = if (appName.isDefined) s"_AppName_${appName.get}" else ""
    -   val AppID = if (appID.isDefined) s"_AppID_${appID.get}" else ""
    -   val AppAttemptID = if (appAttemptID.isDefined) s"_${appAttemptID.get}" 
else ""
    -   val JobID = if (jobID.isDefined) s"_JobID_${jobID.get}" else ""
    -   val StageID = if (stageID.isDefined) s"_StageID_${stageID.get}" else ""
    -   val StageAttemptId = if (stageAttemptId.isDefined) 
s"_${stageAttemptId.get}" else ""
    +   val AppId = if (appId.isDefined) s"_AppId_${appId.get}" else ""
    +   val AppAttemptId = if (appAttemptId.isDefined) 
s"_AttemptId_${appAttemptId.get}" else ""
    --- End diff --
    I don't agree with adding the AttemptId_ string back.  I understand it 
helps readability but its a lot of characters compared to the actual attempt 
id.   I want this thing to be as small as possible as to not add extra overhead 
to the rpc calls.   The strings from your example run are already 112 
characters long 
 if you start getting into many tasks and stages that could reach the default 
128 print to the audit log and get truncated.      
    how about 
SPARK_TASK_application_1474394339641_0006_1_JId_0_SId_0_0_TId_14_0, that is 
only 66 characters.  yes the user may have to look up the format but I think 
that is ok.  If its really that big of an issue parsing this we can change it 
later but I would rather have it smaller and better performance and and have 
all the information (rather then it possibly getting truncated). 
     Really right now I think all you need is the task id and attempt because 
the numbers just increase across jobs and stages, but having the job id and 
stage id would be helpful to find the task id quickly and handles if that every 
changes and we have taskid unique per job or stage.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to