These are executor's logs, not the driver logs. To see this log files, you have to go to executor machines where tasks is running. To see what you will print to stdout or stderr you can either go to the executor machines directly (will store in "stdout" and "stderr" files somewhere in the executor machine) or see through webui
2016-04-29 20:03 GMT+07:00 dev loper <spark...@gmail.com>: > Hi Spark Team, > > I have asked the same question on stack overflow , no luck yet. > > > http://stackoverflow.com/questions/36923949/where-to-find-logs-within-spark-rdd-processing-function-yarn-cluster-mode?noredirect=1#comment61419406_36923949 > > I am running my Spark Application on Yarn Cluster. No matter what I do, I > am not able to get the logs within the RDD function printed . Below you can > find the sample snippet which I have written for the RDD processing > function . I have simplified the code to illustrate the syntax I have used > to write the function. When I am running it locally I am able to see the > logs but not in cluster mode. Neither System.err.println nor the logger > seems to be working. But I could see all my driver logs. I even tried to > log using the Root logger , but it was not working at all within the RDD > processing function .I was desperate to see the log messages so finally I > found a guide to use logger as transient ( > https://www.mapr.com/blog/how-log-apache-spark) ,but event that didn't > help > > class SampleFlatMapFunction implements PairFlatMapFunction > <Tuple2<String,String>,String,String>{ > > private static final long serialVersionUID = 6565656322667L; > transient Logger executorLogger = LogManager.getLogger("sparkExecutor"); > > > private void readObject(java.io.ObjectInputStream in) > throws IOException, ClassNotFoundException { > in.defaultReadObject(); > executorLogger = LogManager.getLogger("sparkExecutor"); > } > @Override > public Iterable<Tuple2<String,String>> call(Tuple2<String, String> tuple) > throws Exception { > > executorLogger.info(" log testing from executorLogger ::"); > System.err.println(" log testing from executorLogger system error > stream "); > > > List<Tuple2<String, String>> updates = new ArrayList<>(); > //process Tuple , expand and add it to list. > return updates; > > } > }; > > My Log4j Configuration is given below > > log4j.appender.console=org.apache.log4j.ConsoleAppender > log4j.appender.console.target=System.err > log4j.appender.console.layout=org.apache.log4j.PatternLayout > log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p > %c{1}: %m%n > > log4j.appender.stdout=org.apache.log4j.ConsoleAppender > log4j.appender.stdout.target=System.out > log4j.appender.stdout.layout=org.apache.log4j.PatternLayout > log4j.appender.stdout.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p > %c{1}: %m%n > > log4j.appender.RollingAppender=org.apache.log4j.DailyRollingFileAppender > log4j.appender.RollingAppender.File=/var/log/spark/spark.log > log4j.appender.RollingAppender.DatePattern='.'yyyy-MM-dd > log4j.appender.RollingAppender.layout=org.apache.log4j.PatternLayout > log4j.appender.RollingAppender.layout.ConversionPattern=[%p] %d %c %M - > %m%n > > log4j.appender.RollingAppenderU=org.apache.log4j.DailyRollingFileAppender > > log4j.appender.RollingAppenderU.File=${spark.yarn.app.container.log.dir}/spark-app.log > log4j.appender.RollingAppenderU.DatePattern='.'yyyy-MM-dd > log4j.appender.RollingAppenderU.layout=org.apache.log4j.PatternLayout > log4j.appender.RollingAppenderU.layout.ConversionPattern=[%p] %d %c %M - > %m%n > > > # By default, everything goes to console and file > log4j.rootLogger=INFO, RollingAppender, console > > # My custom logging goes to another file > log4j.logger.sparkExecutor=INFO, stdout, RollingAppenderU > > > i have tried yarn logs, Spark UI Logs nowhere I could see the log > statements from RDD processing functions . I tried below Approaches but it > didn't work > > yarn logs -applicationId > > I checked even below HDFS path also > > /tmp/logs/ > > > I am running my spark-submit command by passing below arguments, Even then > its not working > > --master yarn --deploy-mode cluster --conf > "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties" > --conf > "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties" > > Can somebody guide me on logging within spark RDD and map functions ? What > am I missing in the above steps ? > > Thanks > > Dev >
