[
https://issues.apache.org/jira/browse/OOZIE-2998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094090#comment-16094090
]
Satish Subhashrao Saley commented on OOZIE-2998:
------------------------------------------------
1. spark-log4j.properties file contains file appender ( This appender caches
the child job id.)
https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java#L202-L203,
we have a file appender, named jobid, which points to file -
/tmp/yarn-local/usercache/saley/appcache/application_1500315465616_1234/container_1500315465616_1234_01_000001/spark-oozie-job_1500315465616_1234.log.
application_1500315465616_1234 is job id for the launcher job.
2. We pass spark-log4j.properties with --files option. Also we add this to java
options to driver and executor
https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java#L234-L240.
3. Now the node on which driver/executor runs (and has different application
id) ,
/tmp/yarn-local/usercache/saley/appcache/application_1500315465616_1234/container_1500315465616_1234_01_000001/spark-oozie-job_1500315465616_1234.log
gets created which is not desired.
Solution would be remove this appender from spark-log4j.properties. Create a
different launcher-log4j.properties file to include this appender.
> Remove file appender from log4.properties for spark executors
> -------------------------------------------------------------
>
> Key: OOZIE-2998
> URL: https://issues.apache.org/jira/browse/OOZIE-2998
> Project: Oozie
> Issue Type: Bug
> Reporter: Satish Subhashrao Saley
> Assignee: Satish Subhashrao Saley
>
> We add a file appender in spark-log4j.properties file. We add this file to
> --files option so that spark executors obeys the oozie.spark.log.level and
> oozie.action.rootlogger.log.level. But the file appender is resulting in
> stale app dirs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
