When the yarn container runs on Hadoop node, the log4j.properties file is taken
from the Hadoop classpath. There could be use cases where user wants to use
custom log4j.properties file.
There are two ways of setting custom log4j.properties in Apex
1. Setting custom log4j.properties at the Application level: User needs to set
following attribute
<property>
<name>dt.attr.CONTAINER_JVM_OPTIONS</name>
<value>-Dlog4j.configuration=custom_log4j.properties</value>
</property>
This will make sure that custom log4j properties is used for all containers
including Application Master.
2. Setting custom log4j.properties at an individual operator level: User needs
to set the following attribute at an operator level
<property>
<name>dt.operator.<OPERATOR_NAME>.attr.JVM_OPTIONS</name>
<value> -Dlog4j.configuration=custom_log4j.properties</value>
</property>
This is set the custom log4j properties only on the container that is hosting
the operator.
Make sure that custom_log4j.properties is part of your application jar and is
located under src/main/resources
Following are examples of how custom log4j properties file should look like
1. Writing to a file
log4j.rootLogger=${hadoop.root.logger} // this is set to INFO / DEBUG, RFA
log4j.appender.RFA=org.apache.log4j.RollingFileAppender
log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c{2} %M -
%m%n
log4j.appender.RFA.File=${hadoop.log.dir}/${hadoop.log.file}
2. Writing to Console
log4j.rootLogger=${hadoop.root.logger} // this is set to INFO / DEBUG, RFA
log4j.appender.RFA=org.apache.log4j.ConsoleAppender
log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c{2} %M -
%m%n
Thanks
- Gaurav
