I am using Spark on Kubernetes from Spark 2.4.3. I have created a log4j.properties file in my local spark/conf directory and modified it so that the console (or, in the case of Kubernetes, the log) only shows warnings and higher (log4j.rootCategory=WARN, console). I then added the command COPY conf /opt/spark/conf to /root/spark/kubernetes/dockerfiles/spark/Dockerfile and built a new container.
However, when I run that under Kubernetes, the program runs successfully but /opt/spark/conf/log4j.properties is not used (I still see the INFO lines when I run kubectl logs <driver pod>). I have tried other things such as explicitly adding a –properties-file to my spark-submit command and even --conf spark.driver.extraJavaOptions=-Dlog4j.configuration=file:///opt/spark/conf/log4j.properties My log4j.properties file is never seen. How do I customize log4j.properties with Kubernetes? Thanks, Dave Jaffe
