bq. --conf "spark.executor.extraJavaOptions=-Dlog4j. configuration=env/dev/log4j-driver.properties"
I think the above may have a typo : you refer to log4j-driver.properties in both arguments. FYI On Wed, Apr 13, 2016 at 8:09 AM, Carlos Rojas Matas <cma...@despegar.com> wrote: > Hi guys, > > I'm trying to enable logging in the executors but with no luck. > > According to the oficial documentation and several blogs, this should be > done passing the > "spark.executor.extraJavaOpts=-Dlog4j.configuration=[my-file]" to the > spark-submit tool. I've tried both sending a reference to a classpath > resource as using the "file:" protocol but nothing happens. Of course in > the later case, I've used the --file option in the command line, although > is not clear where this file is uploaded in the worker machine. > > However, I was able to make it work by setting the properties in the > spark-defaults.conf file pointing to each one of the configurations on the > machine. This approach has a big drawback though: if I change something in > the log4j configuration I need to change it in every machine (and I''m not > sure if restarting is required) which is not what I'm looking for. > > The complete command I'm using is as follows: > > bin/spark-submit --master spark://localhost:7077 --conf > "spark.driver.extraJavaOptions=-Dlog4j.configuration=env/dev/log4j-driver.properties" > --conf > "spark.executor.extraJavaOptions=-Dlog4j.configuration=env/dev/log4j-driver.properties" > --class [my-main-class] [my-jar].jar > > > Both files are in the classpath and are reachable -- already tested with > the driver. > > Any comments will be welcomed. > > Thanks in advance. > -carlos. > >
