Re: Logging in executors

[Carlos Rojas Matas]

Thanks for your response Ted. You're right, there was a typo. I changed it,
now I'm executing:

bin/spark-submit --master spark://localhost:7077 --conf
"spark.driver.extraJavaOptions=-Dlog4j.configuration=env/dev/log4j-driver.properties"
--conf
"spark.executor.extraJavaOptions=-Dlog4j.configuration=env/dev/log4j-executor.properties"
--class....

The content of this file is:

# Set everything to be logged to the console
log4j.rootCategory=INFO, FILE
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p
%c{1}: %m%n

log4j.appender.FILE=org.apache.log4j.RollingFileAppender
log4j.appender.FILE.File=/tmp/executor.log
log4j.appender.FILE.ImmediateFlush=true
log4j.appender.FILE.Threshold=debug
log4j.appender.FILE.Append=true
log4j.appender.FILE.MaxFileSize=100MB
log4j.appender.FILE.MaxBackupIndex=5
log4j.appender.FILE.layout=org.apache.log4j.PatternLayout
log4j.appender.FILE.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p
%c{1}: %m%n

# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark-project.jetty=WARN
log4j.logger.org.spark-project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR
log4j.logger.com.despegar.p13n=DEBUG

# SPARK-9183: Settings to avoid annoying messages when looking up
nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR


Finally, the code on which I'm using logging in the executor is:

def groupAndCount(keys: DStream[(String, List[String])])(handler:
ResultHandler) = {

  val result = keys.reduceByKey((prior, current) => {
    (prior ::: current)
  }).flatMap {
    case (date, keys) =>
      val rs = keys.groupBy(x => x).map(
          obs =>{
            val (d,t) = date.split("@") match {
              case Array(d,t) => (d,t)
            }
            import org.apache.log4j.Logger
            import scala.collection.JavaConverters._
            val logger: Logger = Logger.getRootLogger
            logger.info(s"Metric retrieved $d")
            Metric("PV", d, obs._1, t, obs._2.size)
        }
      )
      rs
  }

  result.foreachRDD((rdd: RDD[Metric], time: Time) => {
    handler(rdd, time)
  })

}


Originally the import and logger object was outside the map function. I'm
also using the root logger just to see if it's working, but nothing gets
logged. I've checked that the property is set correctly on the executor
side through println(System.getProperty("log4j.configuration")) and is OK,
but still not working.

Thanks again,
-carlos.