Hi
In am using an EMR cluster for running my spark jobs, but after the job
finishes logs disappear,
I have added a log4j.properties in my jar, but all the logs still redirects
to EMR resource manager which vanishes after jobs completes, is there a way
i could redirect the logs to a location in file syatem, I am working on
price points and its very critical for me to maintain logs.
Just to add i get following error when my application starts.
java.io.FileNotFoundException: /etc/spark/conf/log4j.properties (No
such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at java.io.FileInputStream.<init>(FileInputStream.java:101)
at
sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
at
sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
at
org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557)
at
org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
at org.apache.spark.Logging$class.initializeLogging(Logging.scala:122)
at
org.apache.spark.Logging$class.initializeIfNecessary(Logging.scala:107)
at org.apache.spark.Logging$class.log(Logging.scala:51)
at
org.apache.spark.deploy.yarn.ApplicationMaster$.log(ApplicationMaster.scala:607)
at
org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:621)
at
org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
