[
https://issues.apache.org/jira/browse/HDFS-17382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
woosuk.ro updated HDFS-17382:
-----------------------------
Description:
In the current Hadoop 3.4 version, the system relies on Log4j 1.x for logging
purposes. This dependency limits the logging functionality, especially when it
comes to rolling log files such as 'hdfs-audit.log'. Rolling log files is
crucial for managing log size and ensuring logs are rotated out over time to
prevent excessive disk space usage. However, the Log4j 1.x version integrated
within Hadoop lacks the necessary capabilities to efficiently handle log
rolling.
This library extends logging capabilities, including more flexible and
configurable log rolling features. By deploying this library, we can enable
advanced rolling strategies such as time-based rolling, size-based rolling, and
compression of rolled logs, which are not supported by the default Log4j 1.x
setup in Hadoop.
The integration of Apache Log4j Extras into Hadoop will significantly improve
log management by allowing for more sophisticated and configurable log rotation
policies. This enhancement is particularly important for maintaining system
performance and reliability, as well as for compliance with log retention
policies.
Although there are plans to upgrade to Log4j 2 in the forthcoming Hadoop 3.5
version, which will inherently solve these issues by providing enhanced logging
features, there is an immediate need to enable advanced log rolling
capabilities in the current and previous versions of Hadoop.
was:
In the current Hadoop 3.3 version, the system relies on Log4j 1.x for logging
purposes. This dependency limits the logging functionality, especially when it
comes to rolling log files such as 'hdfs-audit.log'. Rolling log files is
crucial for managing log size and ensuring logs are rotated out over time to
prevent excessive disk space usage. However, the Log4j 1.x version integrated
within Hadoop lacks the necessary capabilities to efficiently handle log
rolling.
This library extends logging capabilities, including more flexible and
configurable log rolling features. By deploying this library, we can enable
advanced rolling strategies such as time-based rolling, size-based rolling, and
compression of rolled logs, which are not supported by the default Log4j 1.x
setup in Hadoop.
The integration of Apache Log4j Extras into Hadoop will significantly improve
log management by allowing for more sophisticated and configurable log rotation
policies. This enhancement is particularly important for maintaining system
performance and reliability, as well as for compliance with log retention
policies.
Although there are plans to upgrade to Log4j 2 in the forthcoming Hadoop 3.5
version, which will inherently solve these issues by providing enhanced logging
features, there is an immediate need to enable advanced log rolling
capabilities in the current and previous versions of Hadoop.
> Add Apache Log4j Extras Library to Hadoop 3.3 for Enhanced Log Rolling
> Capabilities
> -----------------------------------------------------------------------------------
>
> Key: HDFS-17382
> URL: https://issues.apache.org/jira/browse/HDFS-17382
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 3.4.0
> Reporter: woosuk.ro
> Priority: Minor
> Labels: pull-request-available
> Attachments: image-2024-02-25-00-46-08-841.png
>
>
> In the current Hadoop 3.4 version, the system relies on Log4j 1.x for logging
> purposes. This dependency limits the logging functionality, especially when
> it comes to rolling log files such as 'hdfs-audit.log'. Rolling log files is
> crucial for managing log size and ensuring logs are rotated out over time to
> prevent excessive disk space usage. However, the Log4j 1.x version integrated
> within Hadoop lacks the necessary capabilities to efficiently handle log
> rolling.
> This library extends logging capabilities, including more flexible and
> configurable log rolling features. By deploying this library, we can enable
> advanced rolling strategies such as time-based rolling, size-based rolling,
> and compression of rolled logs, which are not supported by the default Log4j
> 1.x setup in Hadoop.
> The integration of Apache Log4j Extras into Hadoop will significantly improve
> log management by allowing for more sophisticated and configurable log
> rotation policies. This enhancement is particularly important for maintaining
> system performance and reliability, as well as for compliance with log
> retention policies.
> Although there are plans to upgrade to Log4j 2 in the forthcoming Hadoop 3.5
> version, which will inherently solve these issues by providing enhanced
> logging features, there is an immediate need to enable advanced log rolling
> capabilities in the current and previous versions of Hadoop.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
