[
https://issues.apache.org/jira/browse/SPARK-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peng Zhang updated SPARK-2668:
------------------------------
Description:
Assign value of yarn container log directory to java opts "spark.yarn.log.dir",
So user defined log4j.properties can reference this value and write log to YARN
container's log directory.
Otherwise, user defined file appender will only write to container's CWD, and
log files in CWD will not be displayed on YARN UI，and either cannot be
aggregated to HDFS log directory after job finished.
User defined log4j.properties reference example:
{code}
log4j.appender.rolling_file.File = ${spark.yarn.log.dir}/spark.log
{code}
was:
Assign value of yarn container log directory to java opts "spark.yarn.log.dir",
So user defined log4j.properties can reference this value and write log to YARN
container directory.
Otherwise, user defined file appender will only write to container's CWD, and
log files in CWD will not be displayed on YARN UI，and either cannot be
aggregated to HDFS log directory after job finished.
User defined log4j.properties reference example:
{code}
log4j.appender.rolling_file.File = ${spark.yarn.log.dir}/spark.log
{code}
> Add variable of yarn log directory for reference from the log4j configuration
> -----------------------------------------------------------------------------
>
> Key: SPARK-2668
> URL: https://issues.apache.org/jira/browse/SPARK-2668
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 1.0.0
> Reporter: Peng Zhang
>
> Assign value of yarn container log directory to java opts
> "spark.yarn.log.dir", So user defined log4j.properties can reference this
> value and write log to YARN container's log directory.
> Otherwise, user defined file appender will only write to container's CWD, and
> log files in CWD will not be displayed on YARN UI，and either cannot be
> aggregated to HDFS log directory after job finished.
> User defined log4j.properties reference example:
> {code}
> log4j.appender.rolling_file.File = ${spark.yarn.log.dir}/spark.log
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
