Re: Trouble configuring logging
Thanks Dylan - Thanks for your interest in contributing to a solution for
the issue! Let's collaborate on the Jira.
Cheers,
Kevin
On Oct 3, 2022 at 12:59:02, Dylan Klomparens
wrote:
> Pardon me, I forgot to include the link to the JIRA ticket:
> https://issues.apache.org/jira/browse/NIFI-10579
> --
> *From:* Dylan Klomparens
> *Sent:* Monday, October 3, 2022 12:16 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: Trouble configuring logging
>
> Everyone, thank you for the feedback on this topic.
>
> I have created a JIRA Improvement ticket about this topic. I proposed that
> there be a way to disable redirection of standard out and standard error to
> logback.
>
> If there is a positive response from the community and Apache foundation
> members, I will implement this improvement - following the contribution
> guide <https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide>
> - and submit a pull request.
>
>
> --
> *From:* Chris Sampson
> *Sent:* Thursday, September 29, 2022 6:52 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: Trouble configuring logging
>
> We had similar logging configuration issues when using the apache/nifi
> image in a Kubernetes environment, so had to adjust how we log things via
> the logback.xml to make formats a little more consistent, remove some logs
> we didn't (generally) find so useful and hide some things in log files
> within the container because it would cause policy/GDPR issues if it was
> output to StdOut where it could potentially be exposed to other tenants
> within our multi-tenant cloud.
>
> A cutdown/sanitised version of this logback.xml file has been uploaded as
> a gist [1] in case it helps anyone in future - pay attention to the
> comments within the XML if you want to use the file as it's not complete
> (I've removed some duplicated appender definitions where the only
> difference was the appender name and output log file, for example).
>
> We also customised the image with a different startup script that is
> effectively the same as the apache/nifi start.sh but doesn't include the
> "tail" of the "nifi-app.log" file (as we no longer have that file in our
> config and the logback.xml sends everything to StdOut anyway).
>
> In terms of a future approach, I wonder whether taking a similar approach
> to some other applications, like Elasticsearch, would be sensible where
> they include different default configurations in the various distribution
> mechanisms they provide, e.g. RPM vs. Docker Image? Re-invigorating the
> work to make more things configurable via environment variables would also
> be worth considering if someone has time because Spring Boot/Logback can
> definitely handle that, allowing users to then configure their deployment
> with little more than environment variables, etc.
>
>
> [1] https://gist.github.com/ChrisSamo632/813fdfec45f1e0e28c674b133f036811
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.samp...@naimuri.com
>
>
> On Thu, 29 Sept 2022 at 01:21, Kevin Doran wrote:
>
> Yeah, if it helps at all, this is how we work around this in the Apache
> Dockerfile (I knew realize why this method was used rather than a standard
> console appender in logback):
>
> # Continuously provide logs so that 'docker logs' can produce them
> "${NIFI_HOME}/bin/nifi.sh" run &
> nifi_pid="$!"
> tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" &
>
>
> https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129
>
>
> Something to look into. As more NiFi deployments become containerized,
> this may be an area for improvement over time.
>
> Thanks,
> Kevin
>
> On Sep 28, 2022 at 19:40:35, Dylan Klomparens
> wrote:
>
> Mark and Kevin, thank you for your insightful comments. That information
> allowed me to piece together the puzzle. Indeed, anything that is sent to
> standard out subsequently *causes* the log message to come from the the
> org.apache.nifi.StdOut logger in the RunNiFi class. Your description
> allowed me to find the exact line of code that does this
> <https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485>
> .
>
> From my perspective this is an unfortunate design choice for NiFi because
> it does not allow standard out to be redirected easily. (For example,
> flowing onward to Docker, and forwarded to AWS CloudWatch using Docker's
> built-in log driver, in my case). I suppose I'll have to find an
> alternative logback appender class to output the logs to, then find a way
>
Re: Trouble configuring logging
Pardon me, I forgot to include the link to the JIRA ticket:
https://issues.apache.org/jira/browse/NIFI-10579
From: Dylan Klomparens
Sent: Monday, October 3, 2022 12:16 PM
To: users@nifi.apache.org
Subject: Re: Trouble configuring logging
Everyone, thank you for the feedback on this topic.
I have created a JIRA Improvement ticket about this topic. I proposed that
there be a way to disable redirection of standard out and standard error to
logback.
If there is a positive response from the community and Apache foundation
members, I will implement this improvement - following the contribution
guide<https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide> - and
submit a pull request.
From: Chris Sampson
Sent: Thursday, September 29, 2022 6:52 AM
To: users@nifi.apache.org
Subject: Re: Trouble configuring logging
We had similar logging configuration issues when using the apache/nifi image in
a Kubernetes environment, so had to adjust how we log things via the
logback.xml to make formats a little more consistent, remove some logs we
didn't (generally) find so useful and hide some things in log files within the
container because it would cause policy/GDPR issues if it was output to StdOut
where it could potentially be exposed to other tenants within our multi-tenant
cloud.
A cutdown/sanitised version of this logback.xml file has been uploaded as a
gist [1] in case it helps anyone in future - pay attention to the comments
within the XML if you want to use the file as it's not complete (I've removed
some duplicated appender definitions where the only difference was the appender
name and output log file, for example).
We also customised the image with a different startup script that is
effectively the same as the apache/nifi start.sh but doesn't include the "tail"
of the "nifi-app.log" file (as we no longer have that file in our config and
the logback.xml sends everything to StdOut anyway).
In terms of a future approach, I wonder whether taking a similar approach to
some other applications, like Elasticsearch, would be sensible where they
include different default configurations in the various distribution mechanisms
they provide, e.g. RPM vs. Docker Image? Re-invigorating the work to make more
things configurable via environment variables would also be worth considering
if someone has time because Spring Boot/Logback can definitely handle that,
allowing users to then configure their deployment with little more than
environment variables, etc.
[1] https://gist.github.com/ChrisSamo632/813fdfec45f1e0e28c674b133f036811
---
Chris Sampson
IT Consultant
chris.samp...@naimuri.com<mailto:chris.samp...@naimuri.com>
On Thu, 29 Sept 2022 at 01:21, Kevin Doran
mailto:kdo...@apache.org>> wrote:
Yeah, if it helps at all, this is how we work around this in the Apache
Dockerfile (I knew realize why this method was used rather than a standard
console appender in logback):
# Continuously provide logs so that 'docker logs' can produce them
"${NIFI_HOME}/bin/nifi.sh" run &
nifi_pid="$!"
tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" &
https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129
Something to look into. As more NiFi deployments become containerized, this may
be an area for improvement over time.
Thanks,
Kevin
On Sep 28, 2022 at 19:40:35, Dylan Klomparens
mailto:dklompar...@foodallergy.org>> wrote:
Mark and Kevin, thank you for your insightful comments. That information
allowed me to piece together the puzzle. Indeed, anything that is sent to
standard out subsequently causes the log message to come from the the
org.apache.nifi.StdOut logger in the RunNiFi class. Your description allowed me
to find the exact line of code that does
this<https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485>.
From my perspective this is an unfortunate design choice for NiFi because it
does not allow standard out to be redirected easily. (For example, flowing
onward to Docker, and forwarded to AWS CloudWatch using Docker's built-in log
driver, in my case). I suppose I'll have to find an alternative logback
appender class to output the logs to, then find a way to forward them on from
there.
Thanks again for the additional information that put the picture into view.
From: Mark Payne mailto:marka...@hotmail.com>>
Sent: Wednesday, September 28, 2022 10:14 AM
To: users mailto:users@nifi.apache.org>>
Subject: Re: Trouble configuring logging
[EXTERNAL]
This is because of how NiFi is run. When you startup nifi (bin/nifi.sh start)
it doesn’t directly start the NiFi process.
Instead, it starts a different processor, which is called RunNiFi. Th
Re: Trouble configuring logging
Everyone, thank you for the feedback on this topic.
I have created a JIRA Improvement ticket about this topic. I proposed that
there be a way to disable redirection of standard out and standard error to
logback.
If there is a positive response from the community and Apache foundation
members, I will implement this improvement - following the contribution
guide<https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide> - and
submit a pull request.
From: Chris Sampson
Sent: Thursday, September 29, 2022 6:52 AM
To: users@nifi.apache.org
Subject: Re: Trouble configuring logging
We had similar logging configuration issues when using the apache/nifi image in
a Kubernetes environment, so had to adjust how we log things via the
logback.xml to make formats a little more consistent, remove some logs we
didn't (generally) find so useful and hide some things in log files within the
container because it would cause policy/GDPR issues if it was output to StdOut
where it could potentially be exposed to other tenants within our multi-tenant
cloud.
A cutdown/sanitised version of this logback.xml file has been uploaded as a
gist [1] in case it helps anyone in future - pay attention to the comments
within the XML if you want to use the file as it's not complete (I've removed
some duplicated appender definitions where the only difference was the appender
name and output log file, for example).
We also customised the image with a different startup script that is
effectively the same as the apache/nifi start.sh but doesn't include the "tail"
of the "nifi-app.log" file (as we no longer have that file in our config and
the logback.xml sends everything to StdOut anyway).
In terms of a future approach, I wonder whether taking a similar approach to
some other applications, like Elasticsearch, would be sensible where they
include different default configurations in the various distribution mechanisms
they provide, e.g. RPM vs. Docker Image? Re-invigorating the work to make more
things configurable via environment variables would also be worth considering
if someone has time because Spring Boot/Logback can definitely handle that,
allowing users to then configure their deployment with little more than
environment variables, etc.
[1] https://gist.github.com/ChrisSamo632/813fdfec45f1e0e28c674b133f036811
---
Chris Sampson
IT Consultant
chris.samp...@naimuri.com<mailto:chris.samp...@naimuri.com>
On Thu, 29 Sept 2022 at 01:21, Kevin Doran
mailto:kdo...@apache.org>> wrote:
Yeah, if it helps at all, this is how we work around this in the Apache
Dockerfile (I knew realize why this method was used rather than a standard
console appender in logback):
# Continuously provide logs so that 'docker logs' can produce them
"${NIFI_HOME}/bin/nifi.sh" run &
nifi_pid="$!"
tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" &
https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129
Something to look into. As more NiFi deployments become containerized, this may
be an area for improvement over time.
Thanks,
Kevin
On Sep 28, 2022 at 19:40:35, Dylan Klomparens
mailto:dklompar...@foodallergy.org>> wrote:
Mark and Kevin, thank you for your insightful comments. That information
allowed me to piece together the puzzle. Indeed, anything that is sent to
standard out subsequently causes the log message to come from the the
org.apache.nifi.StdOut logger in the RunNiFi class. Your description allowed me
to find the exact line of code that does
this<https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485>.
From my perspective this is an unfortunate design choice for NiFi because it
does not allow standard out to be redirected easily. (For example, flowing
onward to Docker, and forwarded to AWS CloudWatch using Docker's built-in log
driver, in my case). I suppose I'll have to find an alternative logback
appender class to output the logs to, then find a way to forward them on from
there.
Thanks again for the additional information that put the picture into view.
From: Mark Payne mailto:marka...@hotmail.com>>
Sent: Wednesday, September 28, 2022 10:14 AM
To: users mailto:users@nifi.apache.org>>
Subject: Re: Trouble configuring logging
[EXTERNAL]
This is because of how NiFi is run. When you startup nifi (bin/nifi.sh start)
it doesn’t directly start the NiFi process.
Instead, it starts a different processor, which is called RunNiFi. This RunNiFi
process is responsible for doing a lot of things, including monitoring the nifi
process and if it dies restarting it.
Anything written to NiFi’s Standard Out goes to this processor, which then logs
it.
So you’d probably need to update the logging for
Re: Trouble configuring logging
Thanks Chris! This looks very helpful.
In terms of a future approach, I wonder whether taking a similar approach
> to some other applications, like Elasticsearch, would be sensible where
> they include different default configurations in the various distribution
> mechanisms they provide, e.g. RPM vs. Docker Image? Re-invigorating the
> work to make more things configurable via environment variables would also
> be worth considering if someone has time because Spring Boot/Logback can
> definitely handle that, allowing users to then configure their deployment
> with little more than environment variables, etc.
>
Yeah I love that idea. I linked this email thread on our Docker
Improvements page [1], where we have a placeholder for logging
improvements. I actually may have time to make some contributions to the
Docker Hub image in coming months 爛
[1]
https://cwiki.apache.org/confluence/display/NIFI/NiFi+Docker+Container+Improvements
Cheers,
Kevin
On Sep 29, 2022 at 06:52:57, Chris Sampson
wrote:
> We had similar logging configuration issues when using the apache/nifi
> image in a Kubernetes environment, so had to adjust how we log things via
> the logback.xml to make formats a little more consistent, remove some logs
> we didn't (generally) find so useful and hide some things in log files
> within the container because it would cause policy/GDPR issues if it was
> output to StdOut where it could potentially be exposed to other tenants
> within our multi-tenant cloud.
>
> A cutdown/sanitised version of this logback.xml file has been uploaded as
> a gist [1] in case it helps anyone in future - pay attention to the
> comments within the XML if you want to use the file as it's not complete
> (I've removed some duplicated appender definitions where the only
> difference was the appender name and output log file, for example).
>
> We also customised the image with a different startup script that is
> effectively the same as the apache/nifi start.sh but doesn't include the
> "tail" of the "nifi-app.log" file (as we no longer have that file in our
> config and the logback.xml sends everything to StdOut anyway).
>
>
>
>
> [1] https://gist.github.com/ChrisSamo632/813fdfec45f1e0e28c674b133f036811
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.samp...@naimuri.com
>
>
> On Thu, 29 Sept 2022 at 01:21, Kevin Doran wrote:
>
>> Yeah, if it helps at all, this is how we work around this in the Apache
>> Dockerfile (I knew realize why this method was used rather than a standard
>> console appender in logback):
>>
>> # Continuously provide logs so that 'docker logs' can produce them
>> "${NIFI_HOME}/bin/nifi.sh" run &
>> nifi_pid="$!"
>> tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" &
>>
>>
>> https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129
>>
>>
>> Something to look into. As more NiFi deployments become containerized,
>> this may be an area for improvement over time.
>>
>> Thanks,
>> Kevin
>>
>> On Sep 28, 2022 at 19:40:35, Dylan Klomparens <
>> dklompar...@foodallergy.org> wrote:
>>
>>> Mark and Kevin, thank you for your insightful comments. That information
>>> allowed me to piece together the puzzle. Indeed, anything that is sent to
>>> standard out subsequently *causes* the log message to come from the the
>>> org.apache.nifi.StdOut logger in the RunNiFi class. Your description
>>> allowed me to find the exact line of code that does this
>>> <https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485>
>>> .
>>>
>>> From my perspective this is an unfortunate design choice for NiFi
>>> because it does not allow standard out to be redirected easily. (For
>>> example, flowing onward to Docker, and forwarded to AWS CloudWatch using
>>> Docker's built-in log driver, in my case). I suppose I'll have to find an
>>> alternative logback appender class to output the logs to, then find a way
>>> to forward them on from there.
>>>
>>> Thanks again for the additional information that put the picture into
>>> view.
>>> --
>>> *From:* Mark Payne
>>> *Sent:* Wednesday, September 28, 2022 10:14 AM
>>> *To:* users
>>> *Subject:* Re: Trouble configuring logging
>>>
>>>
>>> [EXTERNAL]
>>> This is because of how NiFi is run. When you startup nifi (bin/nifi.sh
>>> start) it doesn’t directly start the NiFi pro
Re: Trouble configuring logging
We had similar logging configuration issues when using the apache/nifi
image in a Kubernetes environment, so had to adjust how we log things via
the logback.xml to make formats a little more consistent, remove some logs
we didn't (generally) find so useful and hide some things in log files
within the container because it would cause policy/GDPR issues if it was
output to StdOut where it could potentially be exposed to other tenants
within our multi-tenant cloud.
A cutdown/sanitised version of this logback.xml file has been uploaded as a
gist [1] in case it helps anyone in future - pay attention to the comments
within the XML if you want to use the file as it's not complete (I've
removed some duplicated appender definitions where the only difference was
the appender name and output log file, for example).
We also customised the image with a different startup script that is
effectively the same as the apache/nifi start.sh but doesn't include the
"tail" of the "nifi-app.log" file (as we no longer have that file in our
config and the logback.xml sends everything to StdOut anyway).
In terms of a future approach, I wonder whether taking a similar approach
to some other applications, like Elasticsearch, would be sensible where
they include different default configurations in the various distribution
mechanisms they provide, e.g. RPM vs. Docker Image? Re-invigorating the
work to make more things configurable via environment variables would also
be worth considering if someone has time because Spring Boot/Logback can
definitely handle that, allowing users to then configure their deployment
with little more than environment variables, etc.
[1] https://gist.github.com/ChrisSamo632/813fdfec45f1e0e28c674b133f036811
---
*Chris Sampson*
IT Consultant
chris.samp...@naimuri.com
On Thu, 29 Sept 2022 at 01:21, Kevin Doran wrote:
> Yeah, if it helps at all, this is how we work around this in the Apache
> Dockerfile (I knew realize why this method was used rather than a standard
> console appender in logback):
>
> # Continuously provide logs so that 'docker logs' can produce them
> "${NIFI_HOME}/bin/nifi.sh" run &
> nifi_pid="$!"
> tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" &
>
>
> https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129
>
>
> Something to look into. As more NiFi deployments become containerized,
> this may be an area for improvement over time.
>
> Thanks,
> Kevin
>
> On Sep 28, 2022 at 19:40:35, Dylan Klomparens
> wrote:
>
>> Mark and Kevin, thank you for your insightful comments. That information
>> allowed me to piece together the puzzle. Indeed, anything that is sent to
>> standard out subsequently *causes* the log message to come from the the
>> org.apache.nifi.StdOut logger in the RunNiFi class. Your description
>> allowed me to find the exact line of code that does this
>> <https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485>
>> .
>>
>> From my perspective this is an unfortunate design choice for NiFi because
>> it does not allow standard out to be redirected easily. (For example,
>> flowing onward to Docker, and forwarded to AWS CloudWatch using Docker's
>> built-in log driver, in my case). I suppose I'll have to find an
>> alternative logback appender class to output the logs to, then find a way
>> to forward them on from there.
>>
>> Thanks again for the additional information that put the picture into
>> view.
>> --
>> *From:* Mark Payne
>> *Sent:* Wednesday, September 28, 2022 10:14 AM
>> *To:* users
>> *Subject:* Re: Trouble configuring logging
>>
>>
>> [EXTERNAL]
>> This is because of how NiFi is run. When you startup nifi (bin/nifi.sh
>> start) it doesn’t directly start the NiFi process.
>> Instead, it starts a different processor, which is called RunNiFi. This
>> RunNiFi process is responsible for doing a lot of things, including
>> monitoring the nifi process and if it dies restarting it.
>> Anything written to NiFi’s Standard Out goes to this processor, which
>> then logs it.
>> So you’d probably need to update the logging for the appender writing to
>> the bootstrap file:
>> > class="ch.qos.logback.core.rolling.RollingFileAppender”>
>>
>> And redirect that to standard out
>>
>> Thanks
>> -Mark
>>
>>
>> On Sep 28, 2022, at 9:48 AM, Kevin Doran wrote:
>>
>> Dylan - I looked into this and am yet unable to offer an explaination.
>> Perhaps others that are familiar with how org.apache.nifi.StdOut
Re: Trouble configuring logging
Yeah, if it helps at all, this is how we work around this in the Apache
Dockerfile (I knew realize why this method was used rather than a standard
console appender in logback):
# Continuously provide logs so that 'docker logs' can produce them
"${NIFI_HOME}/bin/nifi.sh" run &
nifi_pid="$!"
tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" &
https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129
Something to look into. As more NiFi deployments become containerized, this
may be an area for improvement over time.
Thanks,
Kevin
On Sep 28, 2022 at 19:40:35, Dylan Klomparens
wrote:
> Mark and Kevin, thank you for your insightful comments. That information
> allowed me to piece together the puzzle. Indeed, anything that is sent to
> standard out subsequently *causes* the log message to come from the the
> org.apache.nifi.StdOut logger in the RunNiFi class. Your description
> allowed me to find the exact line of code that does this
> <https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485>
> .
>
> From my perspective this is an unfortunate design choice for NiFi because
> it does not allow standard out to be redirected easily. (For example,
> flowing onward to Docker, and forwarded to AWS CloudWatch using Docker's
> built-in log driver, in my case). I suppose I'll have to find an
> alternative logback appender class to output the logs to, then find a way
> to forward them on from there.
>
> Thanks again for the additional information that put the picture into view.
> --
> *From:* Mark Payne
> *Sent:* Wednesday, September 28, 2022 10:14 AM
> *To:* users
> *Subject:* Re: Trouble configuring logging
>
>
> [EXTERNAL]
> This is because of how NiFi is run. When you startup nifi (bin/nifi.sh
> start) it doesn’t directly start the NiFi process.
> Instead, it starts a different processor, which is called RunNiFi. This
> RunNiFi process is responsible for doing a lot of things, including
> monitoring the nifi process and if it dies restarting it.
> Anything written to NiFi’s Standard Out goes to this processor, which then
> logs it.
> So you’d probably need to update the logging for the appender writing to
> the bootstrap file:
> class="ch.qos.logback.core.rolling.RollingFileAppender”>
>
> And redirect that to standard out
>
> Thanks
> -Mark
>
>
> On Sep 28, 2022, at 9:48 AM, Kevin Doran wrote:
>
> Dylan - I looked into this and am yet unable to offer an explaination.
> Perhaps others that are familiar with how org.apache.nifi.StdOut can shed
> some light, or else I will keep digging when I have a block of time. To
> help in my understanding: Which Docker image are you using? Is it the
> apace/nifi image or a custom one, and if custom, can you share the
> Dockerfile?
>
> Thanks,
> Kevin
>
> On Sep 27, 2022 at 10:21:12, Dylan Klomparens
> wrote:
>
> I am attempting to configure logging for NiFi. I have NiFi running in a
> Docker container, which sends all console logs to AWS CloudWatch.
> Therefore, I am configuring NiFi to send all logs to the console.
>
> The problem is, for some reason *all log messages are coming from the
> org.apache.nifi.StdOut logger*. I cannot figure out why, since I would
> like messages to be printed directly from the logger that is receiving them.
>
> It seems like messages are "passing through" loggers, which are ultimately
> printed out from the org.apache.nifi.StdOut logger. Here is an example of
> *one* log message:
> *2022-09-27 10:08:01,849 INFO [NiFi logging handler]
> org.apache.nifi.StdOut* *2022-09-27 10:08:01,848 INFO [pool-6-thread-1]
> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
> Repository*
>
> *Why would every single log message come from the StdOut logger? And how
> can I have logs delivered from the logger they're supposedly originally
> coming from?*
>
> My logback.xml configuration is below for reference.
>
>
>
>
>
>
> true
>
>
>
>
> %date %level [%thread] %logger{40} %msg%n
>
>
>
>
>
> level="INFO"/>
> level="INFO"/>
> name="org.apache.nifi.controller.repository.StandardProcessSession"
> level="WARN" />
>
>
>
> level="ERROR" />
> level="ERROR" />
>
>
> level="ERROR" />
> level="ERROR" />
>
>
>
> level="OFF" />
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> level="ERROR"/>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
Re: Trouble configuring logging
Mark and Kevin, thank you for your insightful comments. That information
allowed me to piece together the puzzle. Indeed, anything that is sent to
standard out subsequently causes the log message to come from the the
org.apache.nifi.StdOut logger in the RunNiFi class. Your description allowed me
to find the exact line of code that does
this<https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485>.
From my perspective this is an unfortunate design choice for NiFi because it
does not allow standard out to be redirected easily. (For example, flowing
onward to Docker, and forwarded to AWS CloudWatch using Docker's built-in log
driver, in my case). I suppose I'll have to find an alternative logback
appender class to output the logs to, then find a way to forward them on from
there.
Thanks again for the additional information that put the picture into view.
From: Mark Payne
Sent: Wednesday, September 28, 2022 10:14 AM
To: users
Subject: Re: Trouble configuring logging
[EXTERNAL]
This is because of how NiFi is run. When you startup nifi (bin/nifi.sh start)
it doesn’t directly start the NiFi process.
Instead, it starts a different processor, which is called RunNiFi. This RunNiFi
process is responsible for doing a lot of things, including monitoring the nifi
process and if it dies restarting it.
Anything written to NiFi’s Standard Out goes to this processor, which then logs
it.
So you’d probably need to update the logging for the appender writing to the
bootstrap file:
mailto:kdo...@apache.org>> wrote:
Dylan - I looked into this and am yet unable to offer an explaination. Perhaps
others that are familiar with how org.apache.nifi.StdOut can shed some light,
or else I will keep digging when I have a block of time. To help in my
understanding: Which Docker image are you using? Is it the apace/nifi image or
a custom one, and if custom, can you share the Dockerfile?
Thanks,
Kevin
On Sep 27, 2022 at 10:21:12, Dylan Klomparens
mailto:dklompar...@foodallergy.org>> wrote:
I am attempting to configure logging for NiFi. I have NiFi running in a Docker
container, which sends all console logs to AWS CloudWatch. Therefore, I am
configuring NiFi to send all logs to the console.
The problem is, for some reason all log messages are coming from the
org.apache.nifi.StdOut logger. I cannot figure out why, since I would like
messages to be printed directly from the logger that is receiving them.
It seems like messages are "passing through" loggers, which are ultimately
printed out from the org.apache.nifi.StdOut logger. Here is an example of one
log message:
2022-09-27 10:08:01,849 INFO [NiFi logging handler] org.apache.nifi.StdOut
2022-09-27 10:08:01,848 INFO [pool-6-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
Repository
Why would every single log message come from the StdOut logger? And how can I
have logs delivered from the logger they're supposedly originally coming from?
My logback.xml configuration is below for reference.
true
%date %level [%thread] %logger{40} %msg%n
Re: Trouble configuring logging
This is because of how NiFi is run. When you startup nifi (bin/nifi.sh start)
it doesn’t directly start the NiFi process.
Instead, it starts a different processor, which is called RunNiFi. This RunNiFi
process is responsible for doing a lot of things, including monitoring the nifi
process and if it dies restarting it.
Anything written to NiFi’s Standard Out goes to this processor, which then logs
it.
So you’d probably need to update the logging for the appender writing to the
bootstrap file:
mailto:kdo...@apache.org>> wrote:
Dylan - I looked into this and am yet unable to offer an explaination. Perhaps
others that are familiar with how org.apache.nifi.StdOut can shed some light,
or else I will keep digging when I have a block of time. To help in my
understanding: Which Docker image are you using? Is it the apace/nifi image or
a custom one, and if custom, can you share the Dockerfile?
Thanks,
Kevin
On Sep 27, 2022 at 10:21:12, Dylan Klomparens
mailto:dklompar...@foodallergy.org>> wrote:
I am attempting to configure logging for NiFi. I have NiFi running in a Docker
container, which sends all console logs to AWS CloudWatch. Therefore, I am
configuring NiFi to send all logs to the console.
The problem is, for some reason all log messages are coming from the
org.apache.nifi.StdOut logger. I cannot figure out why, since I would like
messages to be printed directly from the logger that is receiving them.
It seems like messages are "passing through" loggers, which are ultimately
printed out from the org.apache.nifi.StdOut logger. Here is an example of one
log message:
2022-09-27 10:08:01,849 INFO [NiFi logging handler] org.apache.nifi.StdOut
2022-09-27 10:08:01,848 INFO [pool-6-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
Repository
Why would every single log message come from the StdOut logger? And how can I
have logs delivered from the logger they're supposedly originally coming from?
My logback.xml configuration is below for reference.
true
%date %level [%thread] %logger{40} %msg%n
Re: Trouble configuring logging
Kevin, thank you for taking a look. I am using a custom Dockerfile (and
docker-compose.yaml) I wrote, pasted at the end of this email.
I also ran NiFi natively, without Docker, on my desktop computer and observed
identical results. I initially suspected Docker could somehow be affecting
stdout somehow, but in testing I've not found any evidence of that.
Dockerfile
FROM openjdk:11
# These are environment variables that are not meant to be
# configurable, they are essential to the operation of the container.
ENV NIFI_HOME /opt/nifi/
ENV NIFI_LOG_DIR /persistent-storage/logs
ENV NIFI_OVERRIDE_NIFIENV true
WORKDIR /opt
# Unpack NiFi and Toolkit
COPY nifi.zip /opt
COPY nifi-toolkit.zip /opt
RUN unzip nifi.zip
RUN unzip nifi-toolkit.zip
RUN mv nifi-1.17.0 nifi
RUN mv nifi-toolkit-1.17.0 nifi-toolkit
# Clean out unused files
RUN rm --force nifi.tar.gz nifi-toolkit.tar.gz
RUN rm --force /opt/nifi/bin/*.bat
RUN rm --force /opt/nifi/conf/*
RUN rm --force /opt/nifi-toolkit/*.bat
RUN mkdir /persistent-storage
WORKDIR /opt/nifi
# Set configuration
COPY bootstrap.conf bootstrap-notification-services.xml authorizers.xml
state-management.xml logback.xml /opt/nifi/conf/
COPY initialize.sh /opt/nifi/bin/
COPY postgres-jdbc-driver.jar /opt
COPY snowflake-jdbc-driver.jar /opt
COPY ZIP_codes_to_states.csv /opt
CMD ["/opt/nifi/bin/nifi.sh", "run"]
EXPOSE 8000/tcp
docker-compose.yaml
services:
nifi:
container_name: nifi
build: .
image: nifi
volumes: [/persistent-storage/:/persistent-storage/]
ports: [8000:8000]
environment: [INITIAL_ADMIN_IDENTITY:'*redacted*']
logging:
driver: "awslogs"
options:
awslogs-region: "*redacted*"
awslogs-group: "*redacted*"
awslogs-stream: "All logs"
From: Kevin Doran
Sent: Wednesday, September 28, 2022 9:48 AM
To: users@nifi.apache.org
Subject: Re: Trouble configuring logging
[EXTERNAL]
Dylan - I looked into this and am yet unable to offer an explaination. Perhaps
others that are familiar with how org.apache.nifi.StdOut can shed some light,
or else I will keep digging when I have a block of time. To help in my
understanding: Which Docker image are you using? Is it the apace/nifi image or
a custom one, and if custom, can you share the Dockerfile?
Thanks,
Kevin
On Sep 27, 2022 at 10:21:12, Dylan Klomparens
mailto:dklompar...@foodallergy.org>> wrote:
I am attempting to configure logging for NiFi. I have NiFi running in a Docker
container, which sends all console logs to AWS CloudWatch. Therefore, I am
configuring NiFi to send all logs to the console.
The problem is, for some reason all log messages are coming from the
org.apache.nifi.StdOut logger. I cannot figure out why, since I would like
messages to be printed directly from the logger that is receiving them.
It seems like messages are "passing through" loggers, which are ultimately
printed out from the org.apache.nifi.StdOut logger. Here is an example of one
log message:
2022-09-27 10:08:01,849 INFO [NiFi logging handler] org.apache.nifi.StdOut
2022-09-27 10:08:01,848 INFO [pool-6-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
Repository
Why would every single log message come from the StdOut logger? And how can I
have logs delivered from the logger they're supposedly originally coming from?
My logback.xml configuration is below for reference.
true
%date %level [%thread] %logger{40} %msg%n
Re: Trouble configuring logging
Dylan - I looked into this and am yet unable to offer an explaination.
Perhaps others that are familiar with how org.apache.nifi.StdOut can shed
some light, or else I will keep digging when I have a block of time. To
help in my understanding: Which Docker image are you using? Is it the
apace/nifi image or a custom one, and if custom, can you share the
Dockerfile?
Thanks,
Kevin
On Sep 27, 2022 at 10:21:12, Dylan Klomparens
wrote:
> I am attempting to configure logging for NiFi. I have NiFi running in a
> Docker container, which sends all console logs to AWS CloudWatch.
> Therefore, I am configuring NiFi to send all logs to the console.
>
> The problem is, for some reason *all log messages are coming from the
> org.apache.nifi.StdOut logger*. I cannot figure out why, since I would
> like messages to be printed directly from the logger that is receiving them.
>
> It seems like messages are "passing through" loggers, which are ultimately
> printed out from the org.apache.nifi.StdOut logger. Here is an example of
> *one* log message:
> *2022-09-27 10:08:01,849 INFO [NiFi logging handler]
> org.apache.nifi.StdOut* *2022-09-27 10:08:01,848 INFO [pool-6-thread-1]
> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
> Repository*
>
> *Why would every single log message come from the StdOut logger? And how
> can I have logs delivered from the logger they're supposedly originally
> coming from?*
>
> My logback.xml configuration is below for reference.
>
>
>
>
>
>
> true
>
>
>
>
> %date %level [%thread] %logger{40} %msg%n
>
>
>
>
>
> level="INFO"/>
> level="INFO"/>
> name="org.apache.nifi.controller.repository.StandardProcessSession"
> level="WARN" />
>
>
>
> level="ERROR" />
> level="ERROR" />
>
>
> level="ERROR" />
> level="ERROR" />
>
>
>
> level="OFF" />
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> level="ERROR"/>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
Trouble configuring logging
I am attempting to configure logging for NiFi. I have NiFi running in a Docker
container, which sends all console logs to AWS CloudWatch. Therefore, I am
configuring NiFi to send all logs to the console.
The problem is, for some reason all log messages are coming from the
org.apache.nifi.StdOut logger. I cannot figure out why, since I would like
messages to be printed directly from the logger that is receiving them.
It seems like messages are "passing through" loggers, which are ultimately
printed out from the org.apache.nifi.StdOut logger. Here is an example of one
log message:
2022-09-27 10:08:01,849 INFO [NiFi logging handler] org.apache.nifi.StdOut
2022-09-27 10:08:01,848 INFO [pool-6-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
Repository
Why would every single log message come from the StdOut logger? And how can I
have logs delivered from the logger they're supposedly originally coming from?
My logback.xml configuration is below for reference.
true
%date %level [%thread] %logger{40} %msg%n
