Thanks Dylan - Thanks for your interest in contributing to a solution for the issue! Let's collaborate on the Jira.
Cheers, Kevin On Oct 3, 2022 at 12:59:02, Dylan Klomparens <[email protected]> wrote: > Pardon me, I forgot to include the link to the JIRA ticket: > https://issues.apache.org/jira/browse/NIFI-10579 > ------------------------------ > *From:* Dylan Klomparens <[email protected]> > *Sent:* Monday, October 3, 2022 12:16 PM > *To:* [email protected] <[email protected]> > *Subject:* Re: Trouble configuring logging > > Everyone, thank you for the feedback on this topic. > > I have created a JIRA Improvement ticket about this topic. I proposed that > there be a way to disable redirection of standard out and standard error to > logback. > > If there is a positive response from the community and Apache foundation > members, I will implement this improvement - following the contribution > guide <https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide> > - and submit a pull request. > > > ------------------------------ > *From:* Chris Sampson <[email protected]> > *Sent:* Thursday, September 29, 2022 6:52 AM > *To:* [email protected] <[email protected]> > *Subject:* Re: Trouble configuring logging > > We had similar logging configuration issues when using the apache/nifi > image in a Kubernetes environment, so had to adjust how we log things via > the logback.xml to make formats a little more consistent, remove some logs > we didn't (generally) find so useful and hide some things in log files > within the container because it would cause policy/GDPR issues if it was > output to StdOut where it could potentially be exposed to other tenants > within our multi-tenant cloud. > > A cutdown/sanitised version of this logback.xml file has been uploaded as > a gist [1] in case it helps anyone in future - pay attention to the > comments within the XML if you want to use the file as it's not complete > (I've removed some duplicated appender definitions where the only > difference was the appender name and output log file, for example). > > We also customised the image with a different startup script that is > effectively the same as the apache/nifi start.sh but doesn't include the > "tail" of the "nifi-app.log" file (as we no longer have that file in our > config and the logback.xml sends everything to StdOut anyway). > > In terms of a future approach, I wonder whether taking a similar approach > to some other applications, like Elasticsearch, would be sensible where > they include different default configurations in the various distribution > mechanisms they provide, e.g. RPM vs. Docker Image? Re-invigorating the > work to make more things configurable via environment variables would also > be worth considering if someone has time because Spring Boot/Logback can > definitely handle that, allowing users to then configure their deployment > with little more than environment variables, etc. > > > [1] https://gist.github.com/ChrisSamo632/813fdfec45f1e0e28c674b133f036811 > > --- > *Chris Sampson* > IT Consultant > [email protected] > > > On Thu, 29 Sept 2022 at 01:21, Kevin Doran <[email protected]> wrote: > > Yeah, if it helps at all, this is how we work around this in the Apache > Dockerfile (I knew realize why this method was used rather than a standard > console appender in logback): > > # Continuously provide logs so that 'docker logs' can produce them > "${NIFI_HOME}/bin/nifi.sh" run & > nifi_pid="$!" > tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" & > > > https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129 > > > Something to look into. As more NiFi deployments become containerized, > this may be an area for improvement over time. > > Thanks, > Kevin > > On Sep 28, 2022 at 19:40:35, Dylan Klomparens <[email protected]> > wrote: > > Mark and Kevin, thank you for your insightful comments. That information > allowed me to piece together the puzzle. Indeed, anything that is sent to > standard out subsequently *causes* the log message to come from the the > org.apache.nifi.StdOut logger in the RunNiFi class. Your description > allowed me to find the exact line of code that does this > <https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485> > . > > From my perspective this is an unfortunate design choice for NiFi because > it does not allow standard out to be redirected easily. (For example, > flowing onward to Docker, and forwarded to AWS CloudWatch using Docker's > built-in log driver, in my case). I suppose I'll have to find an > alternative logback appender class to output the logs to, then find a way > to forward them on from there. > > Thanks again for the additional information that put the picture into view. > ------------------------------ > *From:* Mark Payne <[email protected]> > *Sent:* Wednesday, September 28, 2022 10:14 AM > *To:* users <[email protected]> > *Subject:* Re: Trouble configuring logging > > > [EXTERNAL] > This is because of how NiFi is run. When you startup nifi (bin/nifi.sh > start) it doesn’t directly start the NiFi process. > Instead, it starts a different processor, which is called RunNiFi. This > RunNiFi process is responsible for doing a lot of things, including > monitoring the nifi process and if it dies restarting it. > Anything written to NiFi’s Standard Out goes to this processor, which then > logs it. > So you’d probably need to update the logging for the appender writing to > the bootstrap file: > <appender name="BOOTSTRAP_FILE" > class="ch.qos.logback.core.rolling.RollingFileAppender”> > > And redirect that to standard out > > Thanks > -Mark > > > On Sep 28, 2022, at 9:48 AM, Kevin Doran <[email protected]> wrote: > > Dylan - I looked into this and am yet unable to offer an explaination. > Perhaps others that are familiar with how org.apache.nifi.StdOut can shed > some light, or else I will keep digging when I have a block of time. To > help in my understanding: Which Docker image are you using? Is it the > apace/nifi image or a custom one, and if custom, can you share the > Dockerfile? > > Thanks, > Kevin > > On Sep 27, 2022 at 10:21:12, Dylan Klomparens <[email protected]> > wrote: > > I am attempting to configure logging for NiFi. I have NiFi running in a > Docker container, which sends all console logs to AWS CloudWatch. > Therefore, I am configuring NiFi to send all logs to the console. > > The problem is, for some reason *all log messages are coming from the > org.apache.nifi.StdOut logger*. I cannot figure out why, since I would > like messages to be printed directly from the logger that is receiving them. > > It seems like messages are "passing through" loggers, which are ultimately > printed out from the org.apache.nifi.StdOut logger. Here is an example of > *one* log message: > *2022-09-27 10:08:01,849 INFO [NiFi logging handler] > org.apache.nifi.StdOut* *2022-09-27 10:08:01,848 INFO [pool-6-thread-1] > o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile > Repository* > > *Why would every single log message come from the StdOut logger? And how > can I have logs delivered from the logger they're supposedly originally > coming from?* > > My logback.xml configuration is below for reference. > > <?xml version="1.0" encoding="UTF-8"?> > <configuration debug="true"> > <shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook" /> > > <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator"> > <resetJUL>true</resetJUL> > </contextListener> > > <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender"> > <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> > <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> > </encoder> > </appender> > > <logger name="org.apache.nifi" level="INFO"/> > <logger name="org.apache.nifi.processors" level="WARN"/> > <logger name="org.apache.nifi.processors.standard.LogAttribute" > level="INFO"/> > <logger name="org.apache.nifi.processors.standard.LogMessage" > level="INFO"/> > <logger > name="org.apache.nifi.controller.repository.StandardProcessSession" > level="WARN" /> > > <logger name="org.apache.zookeeper.ClientCnxn" level="ERROR" /> > <logger name="org.apache.zookeeper.server.NIOServerCnxn" level="ERROR" /> > <logger name="org.apache.zookeeper.server.NIOServerCnxnFactory" > level="ERROR" /> > <logger name="org.apache.zookeeper.server.NettyServerCnxnFactory" > level="ERROR" /> > <logger name="org.apache.zookeeper.server.quorum" level="ERROR" /> > <logger name="org.apache.zookeeper.ZooKeeper" level="ERROR" /> > <logger name="org.apache.zookeeper.server.PrepRequestProcessor" > level="ERROR" /> > <logger name="org.apache.nifi.controller.reporting.LogComponentStatuses" > level="ERROR" /> > > <logger name="org.apache.calcite.runtime.CalciteException" level="OFF" /> > > <logger name="org.apache.curator.framework.recipes.leader.LeaderSelector" > level="OFF" /> > <logger name="org.apache.curator.ConnectionState" level="OFF" /> > > <!-- Logger for managing logging statements for nifi clusters. --> > <logger name="org.apache.nifi.cluster" level="INFO"/> > > <!-- Logger for logging HTTP requests received by the web server. --> > <logger name="org.apache.nifi.server.JettyServer" level="INFO"/> > > <!-- Logger for managing logging statements for jetty --> > <logger name="org.eclipse.jetty" level="INFO"/> > > <!-- Suppress non-error messages due to excessive logging by class or > library --> > <logger name="org.springframework" level="ERROR"/> > > <!-- Suppress non-error messages due to known warning about redundant path > annotation (NIFI-574) --> > <logger name="org.glassfish.jersey.internal.Errors" level="ERROR"/> > > <!-- Suppress non-error messages due to Jetty AnnotationParser emitting a > large amount of WARNS. Issue described in NIFI-5479. --> > <logger name="org.eclipse.jetty.annotations.AnnotationParser" > level="ERROR"/> > > <!-- Suppress non-error messages from SSHJ which was emitting large > amounts of INFO logs by default --> > <logger name="net.schmizz.sshj" level="WARN" /> > <logger name="com.hierynomus.sshj" level="WARN" /> > > <!-- Suppress non-error messages from SMBJ which was emitting large > amounts of INFO logs by default --> > <logger name="com.hierynomus.smbj" level="WARN" /> > > <!-- Suppress non-error messages from AWS KCL which was emitting large > amounts of INFO logs by default --> > <logger name="com.amazonaws.services.kinesis" level="WARN" /> > > <!-- Suppress non-error messages from Apache Atlas which was emitting > large amounts of INFO logs by default --> > <logger name="org.apache.atlas" level="WARN" /> > > <logger name="org.apache.nifi.web.security.requests" level="INFO" /> > > <!-- > Logger for capturing user events. We do not want to propagate these > log events to the root logger. These messages are only sent to the > user-log appender. > --> > <logger name="org.apache.nifi.web.security" level="INFO" /> > <logger name="org.apache.nifi.web.api.config" level="INFO" /> > <logger name="org.apache.nifi.authorization" level="INFO" /> > <logger name="org.apache.nifi.cluster.authorization" level="INFO" /> > <logger name="org.apache.nifi.web.api.AccessResource" level="INFO" /> > <logger name="org.springframework.security.saml.log" level="WARN" /> > <logger name="org.opensaml" level="WARN" /> > <logger name="org.apache.nifi.web.server.RequestLog" level="INFO" /> > <logger name="org.apache.nifi.bootstrap" level="INFO" /> > <logger name="org.apache.nifi.bootstrap.Command" level="INFO" /> > > <logger name="org.apache.nifi.StdOut" level="INFO" /> > <logger name="org.apache.nifi.StdErr" level="INFO" /> > > <root level="INFO"> > <appender-ref ref="CONSOLE" /> > </root> > > </configuration> > > >
