[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17709132#comment-17709132 ] Viraj Jasani commented on HADOOP-16206: --- A gentle reminder for the review of adding hadoop-logging module on [https://github.com/apache/hadoop/pull/5503] Thanks > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Viraj Jasani >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17709131#comment-17709131 ] Viraj Jasani commented on HADOOP-16206: --- The explanation for breaking Java 11 compilation support is provided on HADOOP-12956. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Viraj Jasani >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17704839#comment-17704839 ] Viraj Jasani commented on HADOOP-16206: --- Sub-task HADOOP-18207 introducing hadoop-logging module is ready for review: [https://github.com/apache/hadoop/pull/5503] Please help with the review, thanks > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Viraj Jasani >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17702846#comment-17702846 ] Viraj Jasani commented on HADOOP-16206: --- FYI, we use some of the core log4j APIs specifically meant to be used for properties based config and log4j2 has this support removed as they were meant to be private usage. Moreover, we also programmatically set monitor interval for dynamic log4j file changes in httpfs server. This is also no longer supported as that is also meant to be kept in config only (properties, xml, json, yaml etc). I have started this thread with log4j dev/user group for the same [https://lists.apache.org/thread/4l7oyk84jpj6br0sn4ymofdcbgfxmtqp] So far, the only recommended solution provided by them is to configure monitor interval in log4j properties only (i.e. user provided custom log4j properties, in case of httpfs Server), and use Configurator API for the properties file specific usage we have. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Viraj Jasani >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692255#comment-17692255 ] Viraj Jasani commented on HADOOP-16206: --- For sub-task HADOOP-18631, PR is ready for review. A short summary: * Migrating Async appenders from code to log4j properties for namenode audit logger as well as datanode/namenode metric loggers * Provided sample log4j properties on how we can configure AsyncAppender to wrap RFA for the loggers * Incompatible change as three hdfs-site configs are no longer in use, they are to be replaced with log4j properties. Deprecated them, added javadoc and log to indicate the log4j replacement rather than relying on the configs. The configs: ** dfs.namenode.audit.log.async ** dfs.namenode.audit.log.async.blocking ** dfs.namenode.audit.log.async.buffer.size * namenode audit logger as well as datanode/namenode metric loggers now use SLF4J logger rather than log4j logger directly Requesting the review on [https://github.com/apache/hadoop/pull/5418] > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17688110#comment-17688110 ] Viraj Jasani commented on HADOOP-16206: --- Sub-task HADOOP-18206 is completed. Thanks for the reviews! I will create next sub-tasks to migrate Async loggers from "dynamically getting added" to log4j.properties as discussed and finalized above. This is necessary because log4j2 no longer supports adding AsyncLogger programatically. This would include "Namenode audit logger" as well as "metric loggers" for Namenode and Datanode. Thanks > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17679967#comment-17679967 ] Viraj Jasani commented on HADOOP-16206: --- PR for the first sub-task HADOOP-18206 is ready for review: [https://github.com/apache/hadoop/pull/5315] Steve has done initial review of the PR (except hdfs part). If any Hdfs expert has some cycles to review the changes, that would be really great. PS: Initial idea was to break HADOOP-18206 down into further sub-tasks and tackle one module at a time, however I realized that it is only going to be more painful given the inter-dependency among the modules. For instance, hdfs, common, tools etc modules cannot be completely cleaned up individually without breaking the build, due to their dependencies on utility classes. Hence, it was decided to cover all modules in one shot with the above PR. Moreover, we cannot completely get rid of commons-logging from classpath because commons-configuration2 and httpclient still need it. Hence, with this PR, the major goals are a) to avoid having to add commons-logging dependency in common, hdfs, yarn, tools, mapreduce modules and b) use it in our codebase. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17631403#comment-17631403 ] shmily commented on HADOOP-16206: - Have you finished this task? Can you briefly introduce the process of using the bridge upgrade? tks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558938#comment-17558938 ] Alex Liu commented on HADOOP-16206: --- Hi [~zhangduo] , Thanks for your help. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557765#comment-17557765 ] Duo Zhang commented on HADOOP-16206: The work is not easy. I haven't found enough time to finish it yet. For security issues, hadoop 3.3.2 or 3.3.3 has already replaced all the log4j dependencies with reload4j dependencies, which should be enough for fixing security issues. You could try to upgrade to these versions. Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557755#comment-17557755 ] Alex Liu commented on HADOOP-16206: --- Hi Team, Great work. We are waiting for this important feature to fix security issues. Does this feature will release with the version 3.4.0, and may I ask when will 3.4.0 release roughly? Because I can't find the release date in the roadmap of Hadoop: [https://cwiki.apache.org/confluence/display/hadoop/Roadmap] Thanks, Alex > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522832#comment-17522832 ] Duo Zhang commented on HADOOP-16206: Thanks [~ste...@apache.org]. Seems no other response, then let me convert this issue to a top level issue, link it with its current parent, and open sub tasks to land the work piece by piece. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17519772#comment-17519772 ] Steve Loughran commented on HADOOP-16206: - wow, that's great work. yes, lets get each piece in on a subtask. we could have a feature branch to get the chain in. we are doing that for s3a input stream work, but we know those changes will be isolated from the rest of the system so most changes to the code won't affect it. Pushing your work in immediately means that you don't have to worry about incompatibility... only everyone else with pending PRs. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17519391#comment-17519391 ] Duo Zhang commented on HADOOP-16206: I've already modified more than 1k lines of code(do not include the log4j.properties change), but maybe only half done. So finally I can provide a patch but I do not think it is easy to review or test. The related code must be changed are: 1. Dynamically changing appenders(add a async appender in front of all the current appenders usually) and log levels. 2. Adding an appender in UTs to capture the log output. 3. Customized appenders, like Log4Json(is it still used?) or Log4jWarningErrorMetricsAppender. 4. In hadoop-kms we will programmingly load the log4j.properties(is it really necessary?) And we still references commons-logging at many places, which should also be purged. I prefer we first discuss and fix the above problems one by one, before finally moving to log4j2, to make the final move a bit easier. For example, I really doubt the value of modifying appenders at runtime, especially adding async appender. This should be done in log4j.properties, and you can also use the -Dhadoop.root.logger=INFO,AsyncAppender to control the appender to use when starting. And for changing log levels and capture log output in UTs, I think we could introduce a hadoop-logging module, provide some logging framework free interfaces to support these operations, so all other modules can just reference these interfaces to avoid depending on the real logging framework directly, then when moving to log4j2, we just need to modify one place, instead of modifying a bunch of classes. If no big concerns, I will file new sub tasks to land the above proposals first. Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17515327#comment-17515327 ] Duo Zhang commented on HADOOP-16206: This was talked before. Please see the above comments. The final decision is to move to log4j2 completely, without using the bridge. Of course we could leave the bridge on classpath for our users if they have their own log4j.properties. Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17513687#comment-17513687 ] Jason Wen commented on HADOOP-16206: We are also working on log4j2 migration based on hadoop 3.2.2. The approach we are taking is to make use of log4j-1.2-api bridge to load the current log4j1 properties file so that they can work with the new log4j2 framework without config change. The hadoop codes are modified to use log4j2 api directly and the bridge is purely used for loading log4j1 config which may includes references of custom appenders based on log4j1 appender API. Another benefit of using the bridge is it will support the backward compatibility case if the client provides custom log4j1 properties. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17512740#comment-17512740 ] Duo Zhang commented on HADOOP-16206: Some updates here. I've already cleaned up all the log4j1 dependenies and start to fix the compile errors. There are still lots of commons-logging dependecies, I've cleaned them too. Another problem is async http request log setup, now we have a method in the code base to add async appender, but for log4j2, it is not easy to switch appender at runtime, the suggested way is to configure in the config file, and use system properties to change the appenders. Can be discussed later. Now I'm working on implement Log4jWarningErrorMetricsAppender based on log4j2. The way to customize appenders are changed a lot in log4j2, I need to learn first. Will report back later when I have progresses. Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500564#comment-17500564 ] Sylwester Lachiewicz commented on HADOOP-16206: --- +1 (nb) Please also review release note for recently releases Log4j 2.17.2 [https://logging.apache.org/log4j/2.x/changes-report.html#a2.17.2] They have implemented many improvements to accept old property files. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500548#comment-17500548 ] Wei-Chiu Chuang commented on HADOOP-16206: -- +1 > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500545#comment-17500545 ] Duo Zhang commented on HADOOP-16206: Will go with the above approach if no other concerns. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[
https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17498910#comment-17498910
]
Duo Zhang commented on HADOOP-16206:
After some investigating, ContainerLogAppender and ContainerRollingLogAppender
are used for container logging. The goal is to limit the maximum size of logs.
ContainerLogAppender will use a in memory queue to hold the tail of the log
messages after exceeding the size limit, while ContainerRollingLogAppender is
just a rolling file appender, we control the total size by setting the size
limit for one file and the number of files to keep.
So for these two appenders, I think maybe we could just leave the rolling file
based one, as they have almost the same ability. And if we choose the rolling
file based one, we do not need to implement a special appender any more, just
use the RollingFileAppender for log4j2 is enough. We can read system properties
in log4j2.properties so it is easy to control where the log.
For TaskLogAppender, it is almost the same with ContainerLogAppender but I
haven't seen where we plan to reference it. Searching the repo, I can only find
references in 0.21.0's release note.
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/site/markdown/release/0.21.0/RELEASENOTES.0.21.0.md
{noformat}
MAPREDUCE-1086 | Major | hadoop commands in streaming tasks are trying to write
to tasktracker's log
This patch makes TT to set HADOOP_ROOT_LOGGER to INFO,TLA by default in the
environment of taskjvm and its children.
MAPREDUCE-1607 | Major | Task controller may not set permissions for a task
cleanup attempt's log directory
Fixed initialization of a task-cleanup attempt's log directory by setting
correct permissions via task-controller. Added new log4j properties
hadoop.tasklog.iscleanup and log4j.appender.TLA.isCleanup to
conf/log4j.properties. Changed the userlogs for a task-cleanup attempt to go
into its own directory instead of the original attempt directory. This is an
incompatible change as old userlogs of cleanup attempt-dirs before this release
will no longer be visible.
{noformat}
I think it provides the same ability with ContainerLogAppender before we have
yarn. So I prefer we just remove it.
For HttpRequestLogAppender, it is just a dummy appender, see the code here
https://github.com/apache/hadoop/blob/5eab9719cbf6b9bddbdb4454a5f8e1ae12495492/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpRequestLog.java#L86
We will replace it with the actual log writer at runtime. There is a related
issue https://issues.apache.org/jira/browse/HADOOP-17526, where we want to
replace it with Slf4jRequestLog, so we could just a normal log4j2 appender to
configure it. So after HADOOP-17526, we could also remove it.
For Log4jWarningErrorMetricsAppender, I think we need to port it to log4j2.
Thanks.
> Migrate from Log4j1 to Log4j2
> -
>
> Key: HADOOP-16206
> URL: https://issues.apache.org/jira/browse/HADOOP-16206
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Akira Ajisaka
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HADOOP-16206-wip.001.patch
>
>
> This sub-task is to remove log4j1 dependency and add log4j2 dependency.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[
https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17498595#comment-17498595
]
Duo Zhang commented on HADOOP-16206:
The progress till now:
I've implemented a converter to convert the log4j1 properties file to log4j2
properties file.
https://github.com/Apache9/TempCode/blob/master/src/main/java/com/github/apache9/log4j2/HadoopLog4j1ConfigurationConverter.java
It will try its best to convert and if it is impossible, will wrap the config
value with ' ' so we can easily grep them out.
And while converting, I found some typos, and also an invalid configuration...
{noformat}
zhangduo@zhangduo-ubuntu:~/hadoop/code$ grep -R log4j.threshhold .
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/resources/log4j-server.properties:log4j.threshhold=ALL
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/conf/yarnservice-log4j.properties:log4j.threshhold=ALL
./hadoop-cloud-storage-project/hadoop-cos/src/test/resources/log4j.properties:log4j.threshhold=ALL
{noformat}
'threshhold' -> 'threshold'
{noformat}
zhangduo@zhangduo-ubuntu:~/hadoop/code$ grep -R log4j.logger= .
./hadoop-tools/hadoop-sls/src/main/sample-conf/log4j.properties:log4j.logger=NONE,
test
{noformat}
log4j.logger is not a valid config way and NONE is also not a valid level?
And after converting all the files, I greped all the '' in the code base.
{noformat}
zhangduo@zhangduo-ubuntu:~/hadoop/code$ grep -R . | grep type
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j2.properties:appender.CLA.type
= org.apache.hadoop.yarn.ContainerLogAppender
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j2.properties:appender.CRLA.type
= org.apache.hadoop.yarn.ContainerRollingLogAppender
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j2.properties:appender.shuffleCLA.type
= org.apache.hadoop.yarn.ContainerLogAppender
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j2.properties:appender.shuffleCRLA.type
= org.apache.hadoop.yarn.ContainerRollingLogAppender
./hadoop-common-project/hadoop-common/src/main/conf/log4j2.properties:appender.TLA.type
= org.apache.hadoop.mapred.TaskLogAppender
./hadoop-common-project/hadoop-common/src/main/conf/log4j2.properties:#appender.namenoderequestlog.type
= org.apache.hadoop.http.HttpRequestLogAppender
./hadoop-common-project/hadoop-common/src/main/conf/log4j2.properties:#appender.datanoderequestlog.type
= org.apache.hadoop.http.HttpRequestLogAppender
./hadoop-common-project/hadoop-common/src/main/conf/log4j2.properties:#appender.resourcemanagerrequestlog.type
= org.apache.hadoop.http.HttpRequestLogAppender
./hadoop-common-project/hadoop-common/src/main/conf/log4j2.properties:#appender.jobhistoryrequestlog.type
= org.apache.hadoop.http.HttpRequestLogAppender
./hadoop-common-project/hadoop-common/src/main/conf/log4j2.properties:#appender.nodemanagerrequestlog.type
= org.apache.hadoop.http.HttpRequestLogAppender
./hadoop-common-project/hadoop-common/src/main/conf/log4j2.properties:appender.EWMA.type
= org.apache.hadoop.yarn.util.Log4jWarningErrorMetricsAppender
{noformat}
All of them are customized appenders. They are
org.apache.hadoop.yarn.ContainerLogAppender
org.apache.hadoop.yarn.ContainerRollingLogAppender
org.apache.hadoop.mapred.TaskLogAppender
org.apache.hadoop.http.HttpRequestLogAppender
org.apache.hadoop.yarn.util.Log4jWarningErrorMetricsAppender
Next I will dig more on whether these appenders are still needed, and if so,
will try to port them to log4j2.
Thanks.
> Migrate from Log4j1 to Log4j2
> -
>
> Key: HADOOP-16206
> URL: https://issues.apache.org/jira/browse/HADOOP-16206
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Akira Ajisaka
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HADOOP-16206-wip.001.patch
>
>
> This sub-task is to remove log4j1 dependency and add log4j2 dependency.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17498538#comment-17498538 ] Duo Zhang commented on HADOOP-16206: This is all the log4j configuration files in our code base. ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/resources/log4j-server.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/conf/yarnservice-log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/main/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/resources/log4j.properties ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/log4j.properties ./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/resources/default-log4j.properties ./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/conf/httpfs-log4j.properties ./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/resources/httpfs-log4j.properties ./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/resources/default-log4j.properties ./hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/resources/log4j.properties ./hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties ./hadoop-hdfs-project/hadoop-hdfs-client/src/test/resources/log4j.properties ./hadoop-cloud-storage-project/hadoop-cos/src/test/resources/log4j.properties ./hadoop-cloud-storage-project/hadoop-huaweicloud/src/test/resources/log4j.properties ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/resources/log4j.properties ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/resources/log4j.properties ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/log4j.properties ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/resources/log4j.properties ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/log4j.properties ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/resources/log4j.properties ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/log4j.properties ./hadoop-common-project/hadoop-auth-examples/src/main/resources/log4j.properties ./hadoop-common-project/hadoop-registry/src/test/resources/log4j.properties ./hadoop-common-project/hadoop-nfs/src/test/resources/log4j.properties ./hadoop-common-project/hadoop-common/src/main/conf/log4j.properties ./hadoop-common-project/hadoop-common/src/test/resources/log4j.properties ./hadoop-common-project/hadoop-kms/src/main/conf/kms-log4j.properties ./hadoop-common-project/hadoop-kms/src/test/resources/log4j.properties ./hadoop-common-project/hadoop-kms/src/test/resources/log4j-kmsaudit.properties ./hadoop-common-project/hadoop-minikdc/src/main/resources/log4j.properties ./hadoop-client-modules/hadoop-client-integration-tests/src/test/resources/log4j.properties ./hadoop-tools/hadoop-distcp/src/test/resources/log4j.properties ./hadoop-tools/hadoop-azure-datalake/src/test/resources/log4j.properties
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17498537#comment-17498537 ] Duo Zhang commented on HADOOP-16206: The log4j 2.17.2 is out, with LOG4J2-3341 in place. I think it is time to start the work here again. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17487504#comment-17487504 ] Duo Zhang commented on HADOOP-16206: Some updates. I've already tried the newest snapshot version of log4j2 which contains LOG4J2-3341 in HBASE-26723 but we still have some problems. Will come back here once HBASE-26723 could be successfully landed. Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17476518#comment-17476518 ] Duo Zhang commented on HADOOP-16206: Discussed with the log4j2 community and they are willing to add support for passing '-Dlog4j.rootLogger'. Filed LOG4J2-3341 and linked it here. Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[
https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17391481#comment-17391481
]
Akira Ajisaka commented on HADOOP-16206:
{quote}So here for me, the only choice is to do breaking change, like what I
have done in HBase. If we all agree, I still try to go with this direction.
{quote}
+1
> Migrate from Log4j1 to Log4j2
> -
>
> Key: HADOOP-16206
> URL: https://issues.apache.org/jira/browse/HADOOP-16206
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Akira Ajisaka
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HADOOP-16206-wip.001.patch
>
>
> This sub-task is to remove log4j1 dependency and add log4j2 dependency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[
https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17378479#comment-17378479
]
Duo Zhang commented on HADOOP-16206:
{quote}
Are there any updates?
{quote}
Not really. I googled a lot of how to do some magics in the log4j configuration
to split log4j.rootLogger but no progress. Another thing maybe we do something
in java at the very beginning of a start up and do the split but it seems that
we can not set the environment variables in a simple way after a process is
started...
So here for me, the only choice is to do breaking change, like what I have done
in HBase. If we all agree, I still try to go with this direction.
Maybe I should start a discussion thread on the mailing list?
Thanks.
> Migrate from Log4j1 to Log4j2
> -
>
> Key: HADOOP-16206
> URL: https://issues.apache.org/jira/browse/HADOOP-16206
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Akira Ajisaka
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HADOOP-16206-wip.001.patch
>
>
> This sub-task is to remove log4j1 dependency and add log4j2 dependency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17378469#comment-17378469 ] Steve Loughran commented on HADOOP-16206: - do it in trunk. It'll complicate the life of those who backport stuff from 3.4 to 3.3, or into internal builds, but that can't hold things back forever. We can get the changes in trunk, let them stabilise, then worry about what to do after that. backporting to branch-3.3 for a switch in 3.3.2 is going to create tension, even if we make the case "this was a transient dependency and our rules give us a get out clause there" > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17378100#comment-17378100 ] Ahmed Hussein commented on HADOOP-16206: Hi [~zhangduo], Are there any updates? > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[
https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350561#comment-17350561
]
Ahmed Hussein commented on HADOOP-16206:
{quote}The problem is about compatibility... I'm sure that some end users will
use their own scripts to start a hadoop cluster, this change may break their
scripts.
{quote}
log4j used to be controlled through the properties files. If the new parameters
maintain the same default behavior (for each individual module), then this
should be acceptable.
{quote}this change may break their scripts.
{quote}
This is a good point. However, this is a part of pulling changes from trunk
into the remote forks. I do not see that a big concern as long as we are not
modifying a release.
Doing that in trunk should be acceptable.
[~aajisaka] and [~weichiu] Do you guys have any thoughts?
> Migrate from Log4j1 to Log4j2
> -
>
> Key: HADOOP-16206
> URL: https://issues.apache.org/jira/browse/HADOOP-16206
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Akira Ajisaka
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HADOOP-16206-wip.001.patch
>
>
> This sub-task is to remove log4j1 dependency and add log4j2 dependency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350474#comment-17350474 ] Duo Zhang commented on HADOOP-16206: I do not think it is about the difficulty, if we choose to do it then I think we could finally get it done. The problem is about compatibility... I'm sure that some end users will use their own scripts to start a hadoop cluster, this change may break their scripts. For HBase, we clearly mention that for a major release, we are allowed to do breaking changes and we will document this out. But hadoop is another story, it is more widely used than HBase so I'm not sure if this is the best choice... And on testing, maybe a possible way to reduce the complexity, is to introduce a common module with a test-jar contains a log4j2.xml, which will be used for all other moduels to do logging when running UTs, as most of the current log4j.properties files under src/test/resources are almost the same. Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[
https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350465#comment-17350465
]
Ahmed Hussein commented on HADOOP-16206:
{quote}So any suggestions here? Should we do the same trick in HBase, or
someone could find a more compatible way?{quote}
Thanks [~zhangduo] for update.
How difficult it is to add those two system properties to the Hadoop entire
build? How easy would it be to switch the level to DEBUG for testing and
debugging purposes?
> Migrate from Log4j1 to Log4j2
> -
>
> Key: HADOOP-16206
> URL: https://issues.apache.org/jira/browse/HADOOP-16206
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Akira Ajisaka
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HADOOP-16206-wip.001.patch
>
>
> This sub-task is to remove log4j1 dependency and add log4j2 dependency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17349575#comment-17349575 ] Duo Zhang commented on HADOOP-16206: So any suggestions here? Should we do the same trick in HBase, or someone could find a more compatible way? Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[
https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17332431#comment-17332431
]
Duo Zhang commented on HADOOP-16206:
The converted log4j2.xml has some problems. The biggest problem is that, there
is no direct mapping for the configuration style
'log4j.rootLogger=INFO,console'.
In log4j2, you have to set level and appender separately.
In HBase, we keep the config in hbase-env.sh the same, but in bin/hbase, we
split the property and pass level and appender with difference system
properties, i.e, two '-Dxxx', to the actual java process.
This is an example:
https://github.com/apache/hbase/blob/a4d954e60645f2c81d53c2081d90ea9c175314dc/conf/log4j2.xml#L76
{code}
{code}
This is where I split the config:
https://github.com/apache/hbase/blob/a4d954e60645f2c81d53c2081d90ea9c175314dc/bin/hbase#L793
{code}
# log4j2 does not support setting log level and appender at once, so we need to
split HBASE_ROOT_LOGGER
HBASE_ROOT_LOGGER=${HBASE_ROOT_LOGGER:-INFO,console}
array=(${HBASE_ROOT_LOGGER//,/ })
HBASE_OPTS="$HBASE_OPTS -Dhbase.root.logger.level=${array[0]}"
HBASE_OPTS="$HBASE_OPTS -Dhbase.root.logger.appender=${array[1]}"
{code}
In hadoop, we also need to find a way to deal with this.
> Migrate from Log4j1 to Log4j2
> -
>
> Key: HADOOP-16206
> URL: https://issues.apache.org/jira/browse/HADOOP-16206
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Akira Ajisaka
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HADOOP-16206-wip.001.patch
>
>
> This sub-task is to remove log4j1 dependency and add log4j2 dependency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331570#comment-17331570 ] Duo Zhang commented on HADOOP-16206: Implemented a simple tool to convert log4j1 properties to log4j2 xml, based on Log4j1ConfigurationConverter. https://github.com/Apache9/TempCode/blob/master/src/main/java/com/github/apache9/log4j2/HadoopLog4j1ConfigurationConverter.java > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17327051#comment-17327051 ] Duo Zhang commented on HADOOP-16206: Oh, good. Let me check the tool. Thanks for the pointer! > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17326683#comment-17326683 ] Ahmed Hussein commented on HADOOP-16206: Thanks [~zhangduo] for the update. Just curious if you you know about [Log4j1ConfigurationConverter.java|https://logging.apache.org/log4j/2.x/log4j-1.2-api/apidocs/org/apache/log4j/config/Log4j1ConfigurationConverter.html]? I haven't used it before, but it is worth it to give it a try before writing a helper tool from scratch. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17326619#comment-17326619 ] Duo Zhang commented on HADOOP-16206: Some updates here, I changed the dependencies, to remove log4j1 dependency and add log4j2 dependency, tried best to follow the old dependency scope(although some of dependency scopes are wrong I believe). Obviously it will cause compile error as we depend on log4j classes directly in our code base. Will fix them later. Another big project is to convert log4j.properties to log4j2.properties, there are a bunch of these config files in our code base, I tried to modify them manually but soon gave up, as it is impossible to make all the copy paste without mistakes... I need to write a simple helper tool to automatically generate the log4j2.properties based on the log4j.properties. Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17320439#comment-17320439 ] Ahmed Hussein commented on HADOOP-16206: [~zhangduo] Sure, Go ahead and give it a try! > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17320152#comment-17320152 ] Duo Zhang commented on HADOOP-16206: Sounds good to me. So [~ahussein] you would like to provide a patch? If not I could also give a try. Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17320140#comment-17320140 ] Ahmed Hussein commented on HADOOP-16206: Thanks [~weichiu] and [~zhangduo] for the suggestions. The approach of breaking this up seems a good idea. I propose a slightly different scheme of breaking-up the migration. Instead of "per-module", the migration would be split into two phases: # Phase-1: Get it to work. Straightforward replacement of log4j ** IMHO, it will be better to aim for log4j2 skipping the bridge way. ** Reviewer only needs to check that the tests pass and the new configurations are not causing inconsistence. ** this implies that suggestions for tuning/enhancements should be noted but not immediately applied. # Phase-2: Post migration. Tuning and optimizations ** After the migration is done, separate Jiras are filed to tune the logging. ** Separate tickets can be issued to address performance evaluations and exploration of other features such as Async, garbage-free, etc.. The approach of two-phases migration will reduce the burden and logically separate between actual migration Vs. addressing suggestions and tuning requests. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[
https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17319437#comment-17319437
]
Duo Zhang commented on HADOOP-16206:
{quote}
Is it possible to break this up? I.e. Hadoop common, HDFS, and MR. Easier to
review/commit
{quote}
Maybe we could migrate to log4j-1.2-api first, and then move up to log4j2
directly module by module? What do you think?
> Migrate from Log4j1 to Log4j2
> -
>
> Key: HADOOP-16206
> URL: https://issues.apache.org/jira/browse/HADOOP-16206
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Akira Ajisaka
>Priority: Major
> Attachments: HADOOP-16206-wip.001.patch
>
>
> This sub-task is to remove log4j1 dependency and add log4j2 dependency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17319008#comment-17319008 ] Wei-Chiu Chuang commented on HADOOP-16206: -- Is it possible to break this up? I.e. Hadoop common, HDFS, and MR. Easier to review/commit > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[
https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317384#comment-17317384
]
Ahmed Hussein commented on HADOOP-16206:
{quote}So you suggest we do a benchmark first to get a clear evidence of
performance gains in log4j2, and then start our work? Or you mean we could just
start the work as you think we'd better fully move to log4j2?{quote}
The reason I was talking about performance evaluation is that quite often,
there is a bias when presenting numbers of a new implementations.
As long as there are different resources acknowledging the [performance gains
of log4j2|https://logging.apache.org/log4j/log4j-2.2/performance.html], then it
should be fine to move forward. There is no need to re-invent the wheel.
I believe the garbage-free option is an interesting feature to use to reduce
the objects allocation. This leads to less Garbage collection events.
After log4j2 is in, we can use heap analysis to evaluate different
configurations like garbage-free options.
I agree with Steve that it is good to know the performance with different log
configurations. This probably can be done separately in a benchmark to get an
approximate estimate of the tradeoffs.
I am little bit hesitated to work on this issue as I have started on replacing
Guava APIs several months ago and it won't make sense to have such two big
migrations on one plate.
> Migrate from Log4j1 to Log4j2
> -
>
> Key: HADOOP-16206
> URL: https://issues.apache.org/jira/browse/HADOOP-16206
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Akira Ajisaka
>Priority: Major
> Attachments: HADOOP-16206-wip.001.patch
>
>
> This sub-task is to remove log4j1 dependency and add log4j2 dependency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17314849#comment-17314849 ] Steve Loughran commented on HADOOP-16206: - I think we should move. The longer it is put off -the harder it gets. But it'd be good to knew the performance changes of * synchronous logging where the log is configured to *not* include line numbers (no stack creation) * async logging I'd like log4j to for better logging of active thread names > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17314385#comment-17314385 ] Duo Zhang commented on HADOOP-16206: [~ahussein] So you suggest we do a benchmark first to get a clear evidence of performance gains in log4j2, and then start our work? Or you mean we could just start the work as you think we'd better fully move to log4j2? Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17311504#comment-17311504 ] Ahmed Hussein commented on HADOOP-16206: Regarding the concerns that the downstream could use the network classes in log4j, those classes can be removed from the jar file without affecting Hadoop. Therefore, Security wise, the effort to migrate is not worthy. If there is clear evidence of performance gains in log4j2, then this will be the real motivation to migrate. While I like the idea that the log4j bridge could reduce the work significantly, I believe that it would be better to fully move to log4j2. I just think that the bridge may not last long given that it is not clear how its performance would compare to pure log4j2 implementation and how long support we get on the long run (i.e., future CVEs, using new JDKs..etc). > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305441#comment-17305441 ] Duo Zhang commented on HADOOP-16206: HBase has already migrated from log4j1 to log4j2 on master branch, and expect to ship it with 3.0.0 release. The most painful thing when migrating is the log4j1 dependency of hadoop if we want to start mini cluster for testing. Even the log4j-1.2-api can not solve all the problem as we extend the log4j1 FileAppender for implementing special appender for container... I would offer my help here but I'm not sure what is the current direction. Do we want to replace all the old log4j1 config files with log4j2 config files, or we just want to use the log4j-1.2-api to switch the logging framework to log4j2 but still retain the old config files? Or we have other plans here? Thanks. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17283028#comment-17283028 ] Morey Straus commented on HADOOP-16206: --- Thank you [~ste...@apache.org], that helps in my justifications. I would offer to assist but I'm not a coder, just a security/compliance monkey. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17283006#comment-17283006 ] Steve Loughran commented on HADOOP-16206: - [~satanicmechanic] do bear in mind that the log4j CVEs are related to the log4j services which listen on the network to collect and aggregate reports. There is no such deployment in the hadoop codebase, so their security risk is more transient "someone downstream may set things up this way". The big barrier to this has always been the incompatible configuration file format; there is now experimental support for the log4j 1.x files to movement becomes more realistic. Do you want to get involved? > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282462#comment-17282462 ] Morey Straus commented on HADOOP-16206: --- log4j1 has 19 open vulns with a high water mark of critical. This really needs to be prioritized. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264708#comment-17264708 ] Pradyumn Agrawal commented on HADOOP-16206: --- [~aajisaka] are you planning to pick this up, or can I start with this one? I see this one is the only pending sub-task for [Inevitable Log4j2 migration via slf4j|https://issues.apache.org/jira/browse/HADOOP-12956] or am I missing something that this sub-task is not being worked on? > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17181033#comment-17181033 ] Akira Ajisaka commented on HADOOP-16206: [~ahussein] Now I don't have time to work on this issue. Sorry for very late response. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17159284#comment-17159284 ] Ahmed Hussein commented on HADOOP-16206: [~aajisaka] What is the status of that patch? Log4j 2.0 will have some performance improvement and it will be easier to have a built in time-based-rolling-retention. Also, did you consider moving to logback? > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17044518#comment-17044518 ] Steve Loughran commented on HADOOP-16206: - this should go in to 3.4; its too big a change for the 3.3 release, especially for those test suites which do get the log4j loggers and play with their log levels > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041350#comment-17041350 ] Wei-Chiu Chuang commented on HADOOP-16206: -- 3.3.0 code freeze is scheduled in March. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040939#comment-17040939 ] Sourabh Sarvotham Parkala commented on HADOOP-16206: [~weichiu], thank you for the reply. Could you please let me know if the .patch attached in the BLI, is the full fix for log4j migration? Also, is there any release date planned for 3.3.0? Thanks Sourabh > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040264#comment-17040264 ] Wei-Chiu Chuang commented on HADOOP-16206: -- Unless there's a big push for this, it is likely going to happen after Hadoop 3.3.0 (3.4.0 maybe?) 2.7 is dead. 2.8 is pretty much dead by now (i sent an EOL discussion thread in the dev mailing lists). Given that this is an incompat change i suspect we won't cherrypick to lower branches at all. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040136#comment-17040136 ] Sourabh Sarvotham Parkala commented on HADOOP-16206: Request for the actual release date with this update. Also, wanted to check if there is a plan to downport the log4j migration till 2.7.x version as well? > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823842#comment-16823842 ] Akira Ajisaka commented on HADOOP-16206: Note: In log4j1, the logger is set by "-Dhadoop.root.logger" and which sets both the appender and log level. In contrast, log4j2 cannot set both the appender and log level at a time. In other words, we need to have a property to set the appender and another property to set the log level. Therefore we need to rewrite/update some shell scripts. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823772#comment-16823772 ] Akira Ajisaka commented on HADOOP-16206: Attached wip.001 patch * Add -Plog4j2 profile to exclude slf4j-log4j12.jar (slf4j -> log4j1 bridge) from classpath * Add log4j2.properties Notice: * The scope of log4j-slf4j-impl (slf4j -> log4j2 bridge) is set to "provided" in all the modules, to avoid the possible leak of log4j-slf4j-impl jar file to the classpath of downstream projects. * Now log4j-slf4j-impl is not in the classpath but in share/tool/lib directory. I'd like to add the bridge under share/common/lib to include in the classpath, but now I don't know how to do this. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16206-wip.001.patch > > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820778#comment-16820778 ] Akira Ajisaka commented on HADOOP-16206: Now I'm trying to create log4j2.properties for Apache Hadoop. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Priority: Major > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16801894#comment-16801894 ] Akira Ajisaka commented on HADOOP-16206: Maybe the target is 3.3 or 3.4. I think this issue takes a few months or more. > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Priority: Major > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798909#comment-16798909 ] Steve Loughran commented on HADOOP-16206: - target is 3.3, I guess? > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Priority: Major > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16206) Migrate from Log4j1 to Log4j2
[ https://issues.apache.org/jira/browse/HADOOP-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798764#comment-16798764 ] Ryu Kobayashi commented on HADOOP-16206: Nice ticket! > Migrate from Log4j1 to Log4j2 > - > > Key: HADOOP-16206 > URL: https://issues.apache.org/jira/browse/HADOOP-16206 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Priority: Major > > This sub-task is to remove log4j1 dependency and add log4j2 dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
