Hi, Provide the following options in spark-defaults.conf and make sure the log4j.properties file is available in driver and executor:
spark.driver.extraJavaOptions -Dlog4j.configuration=file:<path>/log4j.properties spark.executor.extraJavaOptions -Dlog4j.configuration=file: <path>/log4j.properties Regards Surya From: Rohit Menon <rohit.menon2...@gmail.com> Sent: Friday, September 7, 2018 11:02 PM To: user@spark.apache.org Subject: [K8S] Driver and Executor Logging Hello All, We are trying to use a custom appender for Spark driver and executor pods. However, the changes to log4j.properties file in the spark container image are not taking effect. We even tried simpler changes like changing the logging level to DEBUG. Has anyone run into similar issues? or successfully changed logging properties for spark on k8s? Thanks, Rohit
