Hi,
I am running a sample job as below -
public class WordCountExample {
static Logger logger = LoggerFactory.getLogger(WordCountExample.class);
public static void main(String[] args) throws Exception {
final ExecutionEnvironment env = ExecutionEnvironment.
getExecutionEnvironment();
logger.info("Entering application.");
DataSet<String> text = env.fromElements(
"Who's there?",
"I think I hear them. Stand, ho! Who's there?");
List<Integer> elements = new ArrayList<Integer>();
elements.add(0);
DataSet<TestClass> set = env.fromElements(new TestClass(elements));
DataSet<Tuple2<String, Integer>> wordCounts = text
.flatMap(new LineSplitter())
.withBroadcastSet(set, "set")
.groupBy(0)
.sum(1);
wordCounts.print();
logger.info("Processing done");
//env.execute("wordcount job complete");
}
public static class LineSplitter implements FlatMapFunction<String,
Tuple2<String, Integer>> {
static Logger loggerLineSplitter = LoggerFactory.getLogger(LineSplitter.
class);
@Override
public void flatMap(String line, Collector<Tuple2<String, Integer>> out) {
loggerLineSplitter.info("Logger in LineSplitter.flatMap");
for (String word : line.split(" ")) {
out.collect(new Tuple2<String, Integer>(word, 1));
}
}
}
public static class TestClass implements Serializable {
private static final long serialVersionUID = -2932037991574118651L;
static Logger loggerTestClass = LoggerFactory.getLogger(
"WordCountExample.TestClass");
List<Integer> integerList;
public TestClass(List<Integer> integerList){
this.integerList=integerList;
loggerTestClass.info("Logger in TestClass");
}
}
}
When run in IDE I can see the logs from main class i.e. statements like
below in console logs -
13:40:24.459 [main] INFO com.flink.transform.WordCountExample - Entering
application.
13:40:24.486 [main] INFO WordCountExample.TestClass - Logger in TestClass
When run on Yarn with command - flink run -m yarn-cluster -c
com.flink.transform.WordCountExample rt-1.0-jar-with-dependencies.jar
I only see the flatmap logging statements like -
INFO com.flink.transform.WordCountExample$LineSplitter - Logger in
LineSplitter.flatMap
INFO com.flink.transform.WordCountExample$LineSplitter - Logger in
LineSplitter.flatMap
I have checked the jobmanager and taskmanager logs from yarn in EMR.
This is my log4j.properties from EMR cluster
log4j.rootLogger=INFO,file,elastic
# Config ES logging appender
log4j.appender.elastic=com.letfy.log4j.appenders.ElasticSearchClientAppender
log4j.appender.elastic.elasticHost=http://<>:9200
log4j.appender.elastic.hostName=<>
log4j.appender.elastic.applicationName=<>
# more options (see github project for the full list)
log4j.appender.elastic.elasticIndex=<>
log4j.appender.elastic.elasticType=<>
# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS}
%-5p %-60c %x - %m%n
# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
How can I access main driver logs when run on yarn as master.
Thanks,
Hemant
