[
https://issues.apache.org/jira/browse/HIVE-17704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195479#comment-16195479
]
Sahil Takiar commented on HIVE-17704:
-
Ok had some time to work on this today, here is what I found. The {{spark.log}}
file isn't generated locally anymore. I went back to branch-1.1 and couldn't
even it generate it locally there either. On branch-1.1, the file will only be
generated if Spark picks up {{data/conf/spark/log4j.properties}} as the log4j
config file for the Driver and Executors. Hive tries to ensure this happens by
copying the config file into {{SPARK_HOME/conf/}}, but as far as I can tell
that doesn't always work. I used the {{-Dlog4j.debug}} and found out that Spark
will usually pick up the log4j file from some jar on its classpath. Most blog
posts online discussing how to configure log4j for Spark require setting
{{-Dlog4j.configuration=}} in the {{spark.driver.extraJavaOptions}} config. I
tried that locally at it seems to work. I think there are a few other fixes
that require this to work for the master branch. For one, Spark is still on
log4j1.x, but the properties file we are giving it is for log4j2.x. It would be
nice if each executor logged to a different file.
Alternatively, we could just archive everything under
{{itests/qtest-spark/target/sparkOnYarn/sparkOnYarn-logDir}} which contains all
the container logs. Which is probably the easiest approach
> Hive QA isn't collecting spark.log anymore
> --
>
> Key: HIVE-17704
> URL: https://issues.apache.org/jira/browse/HIVE-17704
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>
> Checked a recent Hive QA run, and the {{spark.log}} file isn't being
> collected by ptest (maybe isn't not even being generated anymore, haven't
> checked).
> We should re-enable it, its hard to debug Hive QA runs without the
> {{spark.log}} file.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
