[
https://issues.apache.org/jira/browse/NIFI-10579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dylan Klomparens updated NIFI-10579:
------------------------------------
Description:
h1. Problem statement and context
NiFi intentionally captures and redirects everything sent to standard out and
standard error. It does so in the [RunNiFi class, seen
here|https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1490].
This approach has the benefit of assigning logback logger classification to
output that is sent to standard out and standard error. Thus, messages that are
printed to standard out are labeled as originating from the
"{{org.apache.nifi.StdOut}}" logback logger, and output from the "{{NiFi
logging handler}}" thread. This catches all stray messages that may casually be
output to standard out or standard error for debugging purposes...
This approach also has drawbacks. It does not permit the NiFi administrator to
redirect standard error or standard out easily. For example, if NiFi is
configured to write all logs to standard out, this "encases" *every* log
message as if it were written to the "{{org.apache.nifi.StdOut}}" logger at the
{{INFO}} log level. For example, suppose logback is configured to write all
messages to the console (standard out) with the message format: [date] [log
level] [thread] [logger] [message]. All subsequent output would be "double
wrapped", first being output by the logger it was emitted from, then being
output by the logger handling standard out, and would look like the sample
below. (Red is the "wrapping" logger, blue is the original message).
{color:#DE350B}[date] INFO [NiFi logging handler] org.apache.nifi.StdOut{color}
{color:#0747A6}[date] [original log level] [original thread] [log
message]{color}
This approach has an additional drawback that all log messages emitted from
{{org.apache.nifi.StdOut}} are output at the {{INFO}} log level. Furthermore,
all log messages emitted from {{org.apache.nifi.StdErr}} are emitted at the
{{ERROR}} log level. This secludes the log level from the source logger, making
log filtering more difficult if the log messages passes through standard out or
standard error. Every log message is forcibly set to one log level.
h1. Why this should be improved
It would be beneficial for system administrators to allow redirection of
standard out and standard error. The ability to pipe log messages to the
external logging solution of their choice would be an important improvement.
For example, when NiFi is run in a Docker container, [Docker can send logs
elsewhere using its configurable log
driver|https://docs.docker.com/config/containers/logging/configure/]. This
means Docker is responsible for buffering the logs and forwarding them on to an
external log solution such as syslog, AWS CloudWatch, Fluent, Google Cloud
Logging, Splunk, and others.
It is beneficial to use an external log solution so that logs do not fill up
the disk of the computer running NiFi.
h1. Proposed improvement
There are at least two approaches that would improve NiFi.
Approach #1: remove the redirection of standard out and standard error in the
RunNiFi class. This would be the most disruptive approach from a deployment
perspective, and may affect some existing instances of NiFi. This is the
easiest technical solution and the most simple.
Approach #2: add a configuration parameter to bootstrap.conf to disable
redirection of standard out and standard error. The default would be that
redirection _does_ still occur. This would be the least disruptive approach,
and would not affect existing instances of NiFi unless the system administrator
specifically configures the setting.
h1. Questions and Implementation
* Would the NiFi developer community be amenable to this change?
* If so, which approach is preferred?
If there is a positive response to this improvement, I will follow the
[Contributor
Guide|https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide] and
implement the desired improvement approach, then submit a pull request.
Thank you for your time and attention.
was:
h1. Problem statement and context
NiFi intentionally captures and redirects everything sent to standard out and
standard error. It does so in the [RunNiFi class, seen
here|https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1490].
This approach has the benefit of assigning logback logger classification to
output that is sent to standard out and standard error. Thus, messages that are
printed to standard out are labeled as originating from the
"{{org.apache.nifi.StdOut}}" logback logger, and output from the "{{NiFi
logging handler}}" thread. This catches all stray messages that may casually be
output to standard out or standard error for debugging purposes...
This approach also has drawbacks. It does not permit the NiFi administrator to
redirect standard error or standard out easily. For example, if NiFi is
configured to write all logs to standard out, this "encases" *every* log
message as if it were written to the "{{org.apache.nifi.StdOut}}" logger at the
{{INFO}} log level. For example, suppose logback is configured to write all
messages to the console (standard out) with the message format: [date] [log
level] [thread] [logger] [message]. All subsequent output would be "double
wrapped", first being output by the logger it was emitted from, then being
output by the logger handling standard out, and would look like the sample
below. (Red is the "wrapping" logger, blue is the original message).
{color:#DE350B}[date] INFO [NiFi logging handler] org.apache.nifi.StdOut{color}
{color:#0747A6}[date] [original log level] [original thread] [log
message]{color}
This approach has an additional drawback that all log messages emitted from
{{org.apache.nifi.StdOut}} are output at the {{INFO}} log level. Furthermore,
all log messages emitted from {{org.apache.nifi.StdErr}} are emitted at the
{{ERROR}} log level. This secludes the log level from the source logger, making
log filtering more difficult if the log messages passes through standard out or
standard error. Every log message is forcibly set to one log level.
h1. Why this should be improved
It would be beneficial for system administrators to allow redirection of
standard out and standard error. The ability to pipe log messages to the
external logging solution of their choice would be an important improvement.
For example, when NiFi is run in a Docker container, [Docker can send logs
elsewhere using its configurable log
driver|https://docs.docker.com/config/containers/logging/configure/]. This
means Docker is responsible for buffering the logs and forwarding them on to an
external log solution such as syslog, AWS CloudWatch, Fluent, Google Cloud
Logging, Splunk, and others.
It is beneficial to use an external log solution so that logs do not fill up
the disk of the computer running NiFi.
h1. Proposed improvement
There are at least two approaches that would improve NiFi.
Approach #1: remove the redirection of standard out and standard error in the
RunNiFi class. This would be the most disruptive approach from a deployment
perspective, and may affect some existing instances of NiFi.
Approach #2: add a configuration parameter to bootstrap.conf to disable
redirection of standard out and standard error. The default would be that
redirection _does_ still occur. This would be the least disruptive approach,
and would not affect existing instances of NiFi unless the system administrator
specifically configures the setting.
h1. Questions and Implementation
* Would the NiFi developer community be amenable to this change?
* If so, which approach is preferred?
If there is a positive response to this improvement, I will follow the
[Contributor
Guide|https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide] and
implement the desired improvement approach, then submit a pull request.
Thank you for your time and attention.
> Improve standard out handling
> -----------------------------
>
> Key: NIFI-10579
> URL: https://issues.apache.org/jira/browse/NIFI-10579
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: 1.16.3
> Environment: All
> Reporter: Dylan Klomparens
> Priority: Minor
> Labels: easyfix, newbie
>
> h1. Problem statement and context
> NiFi intentionally captures and redirects everything sent to standard out and
> standard error. It does so in the [RunNiFi class, seen
> here|https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1490].
> This approach has the benefit of assigning logback logger classification to
> output that is sent to standard out and standard error. Thus, messages that
> are printed to standard out are labeled as originating from the
> "{{org.apache.nifi.StdOut}}" logback logger, and output from the "{{NiFi
> logging handler}}" thread. This catches all stray messages that may casually
> be output to standard out or standard error for debugging purposes...
> This approach also has drawbacks. It does not permit the NiFi administrator
> to redirect standard error or standard out easily. For example, if NiFi is
> configured to write all logs to standard out, this "encases" *every* log
> message as if it were written to the "{{org.apache.nifi.StdOut}}" logger at
> the {{INFO}} log level. For example, suppose logback is configured to write
> all messages to the console (standard out) with the message format: [date]
> [log level] [thread] [logger] [message]. All subsequent output would be
> "double wrapped", first being output by the logger it was emitted from, then
> being output by the logger handling standard out, and would look like the
> sample below. (Red is the "wrapping" logger, blue is the original message).
> {color:#DE350B}[date] INFO [NiFi logging handler]
> org.apache.nifi.StdOut{color} {color:#0747A6}[date] [original log level]
> [original thread] [log message]{color}
> This approach has an additional drawback that all log messages emitted from
> {{org.apache.nifi.StdOut}} are output at the {{INFO}} log level. Furthermore,
> all log messages emitted from {{org.apache.nifi.StdErr}} are emitted at the
> {{ERROR}} log level. This secludes the log level from the source logger,
> making log filtering more difficult if the log messages passes through
> standard out or standard error. Every log message is forcibly set to one log
> level.
> h1. Why this should be improved
> It would be beneficial for system administrators to allow redirection of
> standard out and standard error. The ability to pipe log messages to the
> external logging solution of their choice would be an important improvement.
> For example, when NiFi is run in a Docker container, [Docker can send logs
> elsewhere using its configurable log
> driver|https://docs.docker.com/config/containers/logging/configure/]. This
> means Docker is responsible for buffering the logs and forwarding them on to
> an external log solution such as syslog, AWS CloudWatch, Fluent, Google Cloud
> Logging, Splunk, and others.
> It is beneficial to use an external log solution so that logs do not fill up
> the disk of the computer running NiFi.
> h1. Proposed improvement
> There are at least two approaches that would improve NiFi.
> Approach #1: remove the redirection of standard out and standard error in the
> RunNiFi class. This would be the most disruptive approach from a deployment
> perspective, and may affect some existing instances of NiFi. This is the
> easiest technical solution and the most simple.
> Approach #2: add a configuration parameter to bootstrap.conf to disable
> redirection of standard out and standard error. The default would be that
> redirection _does_ still occur. This would be the least disruptive approach,
> and would not affect existing instances of NiFi unless the system
> administrator specifically configures the setting.
> h1. Questions and Implementation
> * Would the NiFi developer community be amenable to this change?
> * If so, which approach is preferred?
> If there is a positive response to this improvement, I will follow the
> [Contributor
> Guide|https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide] and
> implement the desired improvement approach, then submit a pull request.
> Thank you for your time and attention.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
