We had similar logging configuration issues when using the apache/nifi image in a Kubernetes environment, so had to adjust how we log things via the logback.xml to make formats a little more consistent, remove some logs we didn't (generally) find so useful and hide some things in log files within the container because it would cause policy/GDPR issues if it was output to StdOut where it could potentially be exposed to other tenants within our multi-tenant cloud.
A cutdown/sanitised version of this logback.xml file has been uploaded as a gist [1] in case it helps anyone in future - pay attention to the comments within the XML if you want to use the file as it's not complete (I've removed some duplicated appender definitions where the only difference was the appender name and output log file, for example). We also customised the image with a different startup script that is effectively the same as the apache/nifi start.sh but doesn't include the "tail" of the "nifi-app.log" file (as we no longer have that file in our config and the logback.xml sends everything to StdOut anyway). In terms of a future approach, I wonder whether taking a similar approach to some other applications, like Elasticsearch, would be sensible where they include different default configurations in the various distribution mechanisms they provide, e.g. RPM vs. Docker Image? Re-invigorating the work to make more things configurable via environment variables would also be worth considering if someone has time because Spring Boot/Logback can definitely handle that, allowing users to then configure their deployment with little more than environment variables, etc. [1] https://gist.github.com/ChrisSamo632/813fdfec45f1e0e28c674b133f036811 --- *Chris Sampson* IT Consultant [email protected] On Thu, 29 Sept 2022 at 01:21, Kevin Doran <[email protected]> wrote: > Yeah, if it helps at all, this is how we work around this in the Apache > Dockerfile (I knew realize why this method was used rather than a standard > console appender in logback): > > # Continuously provide logs so that 'docker logs' can produce them > "${NIFI_HOME}/bin/nifi.sh" run & > nifi_pid="$!" > tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" & > > > https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129 > > > Something to look into. As more NiFi deployments become containerized, > this may be an area for improvement over time. > > Thanks, > Kevin > > On Sep 28, 2022 at 19:40:35, Dylan Klomparens <[email protected]> > wrote: > >> Mark and Kevin, thank you for your insightful comments. That information >> allowed me to piece together the puzzle. Indeed, anything that is sent to >> standard out subsequently *causes* the log message to come from the the >> org.apache.nifi.StdOut logger in the RunNiFi class. Your description >> allowed me to find the exact line of code that does this >> <https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485> >> . >> >> From my perspective this is an unfortunate design choice for NiFi because >> it does not allow standard out to be redirected easily. (For example, >> flowing onward to Docker, and forwarded to AWS CloudWatch using Docker's >> built-in log driver, in my case). I suppose I'll have to find an >> alternative logback appender class to output the logs to, then find a way >> to forward them on from there. >> >> Thanks again for the additional information that put the picture into >> view. >> ------------------------------ >> *From:* Mark Payne <[email protected]> >> *Sent:* Wednesday, September 28, 2022 10:14 AM >> *To:* users <[email protected]> >> *Subject:* Re: Trouble configuring logging >> >> >> [EXTERNAL] >> This is because of how NiFi is run. When you startup nifi (bin/nifi.sh >> start) it doesn’t directly start the NiFi process. >> Instead, it starts a different processor, which is called RunNiFi. This >> RunNiFi process is responsible for doing a lot of things, including >> monitoring the nifi process and if it dies restarting it. >> Anything written to NiFi’s Standard Out goes to this processor, which >> then logs it. >> So you’d probably need to update the logging for the appender writing to >> the bootstrap file: >> <appender name="BOOTSTRAP_FILE" >> class="ch.qos.logback.core.rolling.RollingFileAppender”> >> >> And redirect that to standard out >> >> Thanks >> -Mark >> >> >> On Sep 28, 2022, at 9:48 AM, Kevin Doran <[email protected]> wrote: >> >> Dylan - I looked into this and am yet unable to offer an explaination. >> Perhaps others that are familiar with how org.apache.nifi.StdOut can shed >> some light, or else I will keep digging when I have a block of time. To >> help in my understanding: Which Docker image are you using? Is it the >> apace/nifi image or a custom one, and if custom, can you share the >> Dockerfile? >> >> Thanks, >> Kevin >> >> On Sep 27, 2022 at 10:21:12, Dylan Klomparens < >> [email protected]> wrote: >> >> I am attempting to configure logging for NiFi. I have NiFi running in a >> Docker container, which sends all console logs to AWS CloudWatch. >> Therefore, I am configuring NiFi to send all logs to the console. >> >> The problem is, for some reason *all log messages are coming from the >> org.apache.nifi.StdOut logger*. I cannot figure out why, since I would >> like messages to be printed directly from the logger that is receiving them. >> >> It seems like messages are "passing through" loggers, which are >> ultimately printed out from the org.apache.nifi.StdOut logger. Here is >> an example of *one* log message: >> *2022-09-27 10:08:01,849 INFO [NiFi logging handler] >> org.apache.nifi.StdOut* *2022-09-27 10:08:01,848 INFO [pool-6-thread-1] >> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile >> Repository* >> >> *Why would every single log message come from the StdOut logger? And how >> can I have logs delivered from the logger they're supposedly originally >> coming from?* >> >> My logback.xml configuration is below for reference. >> >> <?xml version="1.0" encoding="UTF-8"?> >> <configuration debug="true"> >> <shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook" /> >> >> <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator"> >> <resetJUL>true</resetJUL> >> </contextListener> >> >> <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender"> >> <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> >> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> >> </encoder> >> </appender> >> >> <logger name="org.apache.nifi" level="INFO"/> >> <logger name="org.apache.nifi.processors" level="WARN"/> >> <logger name="org.apache.nifi.processors.standard.LogAttribute" >> level="INFO"/> >> <logger name="org.apache.nifi.processors.standard.LogMessage" >> level="INFO"/> >> <logger >> name="org.apache.nifi.controller.repository.StandardProcessSession" >> level="WARN" /> >> >> <logger name="org.apache.zookeeper.ClientCnxn" level="ERROR" /> >> <logger name="org.apache.zookeeper.server.NIOServerCnxn" level="ERROR" /> >> <logger name="org.apache.zookeeper.server.NIOServerCnxnFactory" >> level="ERROR" /> >> <logger name="org.apache.zookeeper.server.NettyServerCnxnFactory" >> level="ERROR" /> >> <logger name="org.apache.zookeeper.server.quorum" level="ERROR" /> >> <logger name="org.apache.zookeeper.ZooKeeper" level="ERROR" /> >> <logger name="org.apache.zookeeper.server.PrepRequestProcessor" >> level="ERROR" /> >> <logger name="org.apache.nifi.controller.reporting.LogComponentStatuses" >> level="ERROR" /> >> >> <logger name="org.apache.calcite.runtime.CalciteException" level="OFF" /> >> >> <logger name="org.apache.curator.framework.recipes.leader.LeaderSelector" >> level="OFF" /> >> <logger name="org.apache.curator.ConnectionState" level="OFF" /> >> >> <!-- Logger for managing logging statements for nifi clusters. --> >> <logger name="org.apache.nifi.cluster" level="INFO"/> >> >> <!-- Logger for logging HTTP requests received by the web server. --> >> <logger name="org.apache.nifi.server.JettyServer" level="INFO"/> >> >> <!-- Logger for managing logging statements for jetty --> >> <logger name="org.eclipse.jetty" level="INFO"/> >> >> <!-- Suppress non-error messages due to excessive logging by class or >> library --> >> <logger name="org.springframework" level="ERROR"/> >> >> <!-- Suppress non-error messages due to known warning about redundant >> path annotation (NIFI-574) --> >> <logger name="org.glassfish.jersey.internal.Errors" level="ERROR"/> >> >> <!-- Suppress non-error messages due to Jetty AnnotationParser emitting a >> large amount of WARNS. Issue described in NIFI-5479. --> >> <logger name="org.eclipse.jetty.annotations.AnnotationParser" >> level="ERROR"/> >> >> <!-- Suppress non-error messages from SSHJ which was emitting large >> amounts of INFO logs by default --> >> <logger name="net.schmizz.sshj" level="WARN" /> >> <logger name="com.hierynomus.sshj" level="WARN" /> >> >> <!-- Suppress non-error messages from SMBJ which was emitting large >> amounts of INFO logs by default --> >> <logger name="com.hierynomus.smbj" level="WARN" /> >> >> <!-- Suppress non-error messages from AWS KCL which was emitting large >> amounts of INFO logs by default --> >> <logger name="com.amazonaws.services.kinesis" level="WARN" /> >> >> <!-- Suppress non-error messages from Apache Atlas which was emitting >> large amounts of INFO logs by default --> >> <logger name="org.apache.atlas" level="WARN" /> >> >> <logger name="org.apache.nifi.web.security.requests" level="INFO" /> >> >> <!-- >> Logger for capturing user events. We do not want to propagate these >> log events to the root logger. These messages are only sent to the >> user-log appender. >> --> >> <logger name="org.apache.nifi.web.security" level="INFO" /> >> <logger name="org.apache.nifi.web.api.config" level="INFO" /> >> <logger name="org.apache.nifi.authorization" level="INFO" /> >> <logger name="org.apache.nifi.cluster.authorization" level="INFO" /> >> <logger name="org.apache.nifi.web.api.AccessResource" level="INFO" /> >> <logger name="org.springframework.security.saml.log" level="WARN" /> >> <logger name="org.opensaml" level="WARN" /> >> <logger name="org.apache.nifi.web.server.RequestLog" level="INFO" /> >> <logger name="org.apache.nifi.bootstrap" level="INFO" /> >> <logger name="org.apache.nifi.bootstrap.Command" level="INFO" /> >> >> <logger name="org.apache.nifi.StdOut" level="INFO" /> >> <logger name="org.apache.nifi.StdErr" level="INFO" /> >> >> <root level="INFO"> >> <appender-ref ref="CONSOLE" /> >> </root> >> >> </configuration> >> >> >>
