[jira] [Assigned] (OOZIE-3135) Configure log4j2 in SqoopMain
[ https://issues.apache.org/jira/browse/OOZIE-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julia Kinga Marton reassigned OOZIE-3135: - Assignee: (was: Julia Kinga Marton) > Configure log4j2 in SqoopMain > - > > Key: OOZIE-3135 > URL: https://issues.apache.org/jira/browse/OOZIE-3135 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0b1 >Reporter: Attila Sasvari >Priority: Major > Attachments: OOZIE-3135-001.patch > > > In Hadoop 3, MAPREDUCE-6983 switched to use slfj4 & log4j2 in > {{org.apache.hadoop.mapreduce.Job}} (that prints out MR job id-s needed for > Oozie). We need to setup log4j accordingly (it is also related to > HADOOP-12956). > Without proper configuration in the Sqoop action, we won't be able to get > external job id-s (SqoopActionExecutor unit tests and real action would be > also affected). > > [The API for Log4j 2 is not compatible with Log4j > 1.x|https://logging.apache.org/log4j/2.x/], but we will need to support both > hadoop 2 and hadoop 3 profiles for a while. > We could use reflection to determine the type of the logger object in > {{org.apache.hadoop.mapreduce.Job}} and configure log4j settings based on it, > but there might be a better way. > For example we could do something like this: > - add a new method for configuring log4j2: > {code} > private String setUpSqoopLog4J2(final String rootLogLevel) throws > IOException { > System.out.println("Setting up log4j2"); > final String logFile = getSqoopLogFile(); > final File log4j2Xml = new File(SQOOP_LOG4J2_XML); > try (Writer writer = new FileWriter(log4j2Xml)) > { > final String logj2SettingsXml = " encoding=\"UTF-8\"?>\n" + > "\n" + > "\n" + > " target=\"SYSTEM_OUT\">\n" + > " [%t] %-5level %logger{36} - %msg%n\"/>\n" + > "\n" + > " "\"> \n" + > " [%t] %-5level %logger{36} - %msg%n\"/>\n" + > " \n" + > "\n" + > "\n" + > " "\">\n" + > "\n" + > "\n" + > "\n" + > "\n" + > ""; > writer.write(logj2SettingsXml); > } > System.out.printf("log4j2 configuration file created at %s%n", > log4j2Xml.getAbsolutePath()); > final LoggerContext context = (LoggerContext) > LogManager.getContext(false); > context.setConfigLocation(log4j2Xml.toURI()); // forces log4j2 > reconfiguration > return logFile; > } > {code} > and call it in the {{run()}} method if the mapreduce client is using slf4j > for logging: > {code} > String logFile; > // MAPREDUCE-6983 switches to slfj4 & log4j2. Need to setup log4j > accordingly > if > (org.apache.hadoop.mapreduce.Job.class.getDeclaredField("LOG").getType(). > isAssignableFrom(org.slf4j.Logger.class)) { > logFile = setUpSqoopLog4J2(rootLogLevel); > } > else { > logFile = setUpSqoopLog4J(rootLogLevel, logLevel); > } > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (OOZIE-3135) Configure log4j2 in SqoopMain
[ https://issues.apache.org/jira/browse/OOZIE-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julia Kinga Marton reassigned OOZIE-3135: - Assignee: Julia Kinga Marton (was: Attila Sasvari) > Configure log4j2 in SqoopMain > - > > Key: OOZIE-3135 > URL: https://issues.apache.org/jira/browse/OOZIE-3135 > Project: Oozie > Issue Type: Bug >Affects Versions: 5.0.0b1 >Reporter: Attila Sasvari >Assignee: Julia Kinga Marton >Priority: Major > Attachments: OOZIE-3135-001.patch > > > In Hadoop 3, MAPREDUCE-6983 switched to use slfj4 & log4j2 in > {{org.apache.hadoop.mapreduce.Job}} (that prints out MR job id-s needed for > Oozie). We need to setup log4j accordingly (it is also related to > HADOOP-12956). > Without proper configuration in the Sqoop action, we won't be able to get > external job id-s (SqoopActionExecutor unit tests and real action would be > also affected). > > [The API for Log4j 2 is not compatible with Log4j > 1.x|https://logging.apache.org/log4j/2.x/], but we will need to support both > hadoop 2 and hadoop 3 profiles for a while. > We could use reflection to determine the type of the logger object in > {{org.apache.hadoop.mapreduce.Job}} and configure log4j settings based on it, > but there might be a better way. > For example we could do something like this: > - add a new method for configuring log4j2: > {code} > private String setUpSqoopLog4J2(final String rootLogLevel) throws > IOException { > System.out.println("Setting up log4j2"); > final String logFile = getSqoopLogFile(); > final File log4j2Xml = new File(SQOOP_LOG4J2_XML); > try (Writer writer = new FileWriter(log4j2Xml)) > { > final String logj2SettingsXml = " encoding=\"UTF-8\"?>\n" + > "\n" + > "\n" + > " target=\"SYSTEM_OUT\">\n" + > " [%t] %-5level %logger{36} - %msg%n\"/>\n" + > "\n" + > " "\"> \n" + > " [%t] %-5level %logger{36} - %msg%n\"/>\n" + > " \n" + > "\n" + > "\n" + > " "\">\n" + > "\n" + > "\n" + > "\n" + > "\n" + > ""; > writer.write(logj2SettingsXml); > } > System.out.printf("log4j2 configuration file created at %s%n", > log4j2Xml.getAbsolutePath()); > final LoggerContext context = (LoggerContext) > LogManager.getContext(false); > context.setConfigLocation(log4j2Xml.toURI()); // forces log4j2 > reconfiguration > return logFile; > } > {code} > and call it in the {{run()}} method if the mapreduce client is using slf4j > for logging: > {code} > String logFile; > // MAPREDUCE-6983 switches to slfj4 & log4j2. Need to setup log4j > accordingly > if > (org.apache.hadoop.mapreduce.Job.class.getDeclaredField("LOG").getType(). > isAssignableFrom(org.slf4j.Logger.class)) { > logFile = setUpSqoopLog4J2(rootLogLevel); > } > else { > logFile = setUpSqoopLog4J(rootLogLevel, logLevel); > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OOZIE-3135) Configure log4j2 in SqoopMain
[ https://issues.apache.org/jira/browse/OOZIE-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari reassigned OOZIE-3135: - Assignee: Attila Sasvari > Configure log4j2 in SqoopMain > - > > Key: OOZIE-3135 > URL: https://issues.apache.org/jira/browse/OOZIE-3135 > Project: Oozie > Issue Type: Bug >Affects Versions: 5.0.0b1 >Reporter: Attila Sasvari >Assignee: Attila Sasvari > Attachments: OOZIE-3135-001.patch > > > In Hadoop 3, MAPREDUCE-6983 switched to use slfj4 & log4j2 in > {{org.apache.hadoop.mapreduce.Job}} (that prints out MR job id-s needed for > Oozie). We need to setup log4j accordingly (it is also related to > HADOOP-12956). > Without proper configuration in the Sqoop action, we won't be able to get > external job id-s (SqoopActionExecutor unit tests and real action would be > also affected). > > [The API for Log4j 2 is not compatible with Log4j > 1.x|https://logging.apache.org/log4j/2.x/], but we will need to support both > hadoop 2 and hadoop 3 profiles for a while. > We could use reflection to determine the type of the logger object in > {{org.apache.hadoop.mapreduce.Job}} and configure log4j settings based on it, > but there might be a better way. > For example we could do something like this: > - add a new method for configuring log4j2: > {code} > private String setUpSqoopLog4J2(final String rootLogLevel) throws > IOException { > System.out.println("Setting up log4j2"); > final String logFile = getSqoopLogFile(); > final File log4j2Xml = new File(SQOOP_LOG4J2_XML); > try (Writer writer = new FileWriter(log4j2Xml)) > { > final String logj2SettingsXml = " encoding=\"UTF-8\"?>\n" + > "\n" + > "\n" + > " target=\"SYSTEM_OUT\">\n" + > " [%t] %-5level %logger{36} - %msg%n\"/>\n" + > "\n" + > " "\"> \n" + > " [%t] %-5level %logger{36} - %msg%n\"/>\n" + > " \n" + > "\n" + > "\n" + > " "\">\n" + > "\n" + > "\n" + > "\n" + > "\n" + > ""; > writer.write(logj2SettingsXml); > } > System.out.printf("log4j2 configuration file created at %s%n", > log4j2Xml.getAbsolutePath()); > final LoggerContext context = (LoggerContext) > LogManager.getContext(false); > context.setConfigLocation(log4j2Xml.toURI()); // forces log4j2 > reconfiguration > return logFile; > } > {code} > and call it in the {{run()}} method if the mapreduce client is using slf4j > for logging: > {code} > String logFile; > // MAPREDUCE-6983 switches to slfj4 & log4j2. Need to setup log4j > accordingly > if > (org.apache.hadoop.mapreduce.Job.class.getDeclaredField("LOG").getType(). > isAssignableFrom(org.slf4j.Logger.class)) { > logFile = setUpSqoopLog4J2(rootLogLevel); > } > else { > logFile = setUpSqoopLog4J(rootLogLevel, logLevel); > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)