Thanks Chris! This looks very helpful. In terms of a future approach, I wonder whether taking a similar approach > to some other applications, like Elasticsearch, would be sensible where > they include different default configurations in the various distribution > mechanisms they provide, e.g. RPM vs. Docker Image? Re-invigorating the > work to make more things configurable via environment variables would also > be worth considering if someone has time because Spring Boot/Logback can > definitely handle that, allowing users to then configure their deployment > with little more than environment variables, etc. >
Yeah I love that idea. I linked this email thread on our Docker Improvements page [1], where we have a placeholder for logging improvements. I actually may have time to make some contributions to the Docker Hub image in coming months 🤞 [1] https://cwiki.apache.org/confluence/display/NIFI/NiFi+Docker+Container+Improvements Cheers, Kevin On Sep 29, 2022 at 06:52:57, Chris Sampson <[email protected]> wrote: > We had similar logging configuration issues when using the apache/nifi > image in a Kubernetes environment, so had to adjust how we log things via > the logback.xml to make formats a little more consistent, remove some logs > we didn't (generally) find so useful and hide some things in log files > within the container because it would cause policy/GDPR issues if it was > output to StdOut where it could potentially be exposed to other tenants > within our multi-tenant cloud. > > A cutdown/sanitised version of this logback.xml file has been uploaded as > a gist [1] in case it helps anyone in future - pay attention to the > comments within the XML if you want to use the file as it's not complete > (I've removed some duplicated appender definitions where the only > difference was the appender name and output log file, for example). > > We also customised the image with a different startup script that is > effectively the same as the apache/nifi start.sh but doesn't include the > "tail" of the "nifi-app.log" file (as we no longer have that file in our > config and the logback.xml sends everything to StdOut anyway). > > > > > [1] https://gist.github.com/ChrisSamo632/813fdfec45f1e0e28c674b133f036811 > > --- > *Chris Sampson* > IT Consultant > [email protected] > > > On Thu, 29 Sept 2022 at 01:21, Kevin Doran <[email protected]> wrote: > >> Yeah, if it helps at all, this is how we work around this in the Apache >> Dockerfile (I knew realize why this method was used rather than a standard >> console appender in logback): >> >> # Continuously provide logs so that 'docker logs' can produce them >> "${NIFI_HOME}/bin/nifi.sh" run & >> nifi_pid="$!" >> tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" & >> >> >> https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129 >> >> >> Something to look into. As more NiFi deployments become containerized, >> this may be an area for improvement over time. >> >> Thanks, >> Kevin >> >> On Sep 28, 2022 at 19:40:35, Dylan Klomparens < >> [email protected]> wrote: >> >>> Mark and Kevin, thank you for your insightful comments. That information >>> allowed me to piece together the puzzle. Indeed, anything that is sent to >>> standard out subsequently *causes* the log message to come from the the >>> org.apache.nifi.StdOut logger in the RunNiFi class. Your description >>> allowed me to find the exact line of code that does this >>> <https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485> >>> . >>> >>> From my perspective this is an unfortunate design choice for NiFi >>> because it does not allow standard out to be redirected easily. (For >>> example, flowing onward to Docker, and forwarded to AWS CloudWatch using >>> Docker's built-in log driver, in my case). I suppose I'll have to find an >>> alternative logback appender class to output the logs to, then find a way >>> to forward them on from there. >>> >>> Thanks again for the additional information that put the picture into >>> view. >>> ------------------------------ >>> *From:* Mark Payne <[email protected]> >>> *Sent:* Wednesday, September 28, 2022 10:14 AM >>> *To:* users <[email protected]> >>> *Subject:* Re: Trouble configuring logging >>> >>> >>> [EXTERNAL] >>> This is because of how NiFi is run. When you startup nifi (bin/nifi.sh >>> start) it doesn’t directly start the NiFi process. >>> Instead, it starts a different processor, which is called RunNiFi. This >>> RunNiFi process is responsible for doing a lot of things, including >>> monitoring the nifi process and if it dies restarting it. >>> Anything written to NiFi’s Standard Out goes to this processor, which >>> then logs it. >>> So you’d probably need to update the logging for the appender writing to >>> the bootstrap file: >>> <appender name="BOOTSTRAP_FILE" >>> class="ch.qos.logback.core.rolling.RollingFileAppender”> >>> >>> And redirect that to standard out >>> >>> Thanks >>> -Mark >>> >>> >>> On Sep 28, 2022, at 9:48 AM, Kevin Doran <[email protected]> wrote: >>> >>> Dylan - I looked into this and am yet unable to offer an explaination. >>> Perhaps others that are familiar with how org.apache.nifi.StdOut can shed >>> some light, or else I will keep digging when I have a block of time. To >>> help in my understanding: Which Docker image are you using? Is it the >>> apace/nifi image or a custom one, and if custom, can you share the >>> Dockerfile? >>> >>> Thanks, >>> Kevin >>> >>> On Sep 27, 2022 at 10:21:12, Dylan Klomparens < >>> [email protected]> wrote: >>> >>> I am attempting to configure logging for NiFi. I have NiFi running in a >>> Docker container, which sends all console logs to AWS CloudWatch. >>> Therefore, I am configuring NiFi to send all logs to the console. >>> >>> The problem is, for some reason *all log messages are coming from the >>> org.apache.nifi.StdOut logger*. I cannot figure out why, since I would >>> like messages to be printed directly from the logger that is receiving them. >>> >>> It seems like messages are "passing through" loggers, which are >>> ultimately printed out from the org.apache.nifi.StdOut logger. Here is >>> an example of *one* log message: >>> *2022-09-27 10:08:01,849 INFO [NiFi logging handler] >>> org.apache.nifi.StdOut* *2022-09-27 10:08:01,848 INFO [pool-6-thread-1] >>> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile >>> Repository* >>> >>> *Why would every single log message come from the StdOut logger? And how >>> can I have logs delivered from the logger they're supposedly originally >>> coming from?* >>> >>> My logback.xml configuration is below for reference. >>> >>> <?xml version="1.0" encoding="UTF-8"?> >>> <configuration debug="true"> >>> <shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook" /> >>> >>> <contextListener >>> class="ch.qos.logback.classic.jul.LevelChangePropagator"> >>> <resetJUL>true</resetJUL> >>> </contextListener> >>> >>> <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender"> >>> <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> >>> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> >>> </encoder> >>> </appender> >>> >>> <logger name="org.apache.nifi" level="INFO"/> >>> <logger name="org.apache.nifi.processors" level="WARN"/> >>> <logger name="org.apache.nifi.processors.standard.LogAttribute" >>> level="INFO"/> >>> <logger name="org.apache.nifi.processors.standard.LogMessage" >>> level="INFO"/> >>> <logger >>> name="org.apache.nifi.controller.repository.StandardProcessSession" >>> level="WARN" /> >>> >>> <logger name="org.apache.zookeeper.ClientCnxn" level="ERROR" /> >>> <logger name="org.apache.zookeeper.server.NIOServerCnxn" level="ERROR" /> >>> <logger name="org.apache.zookeeper.server.NIOServerCnxnFactory" >>> level="ERROR" /> >>> <logger name="org.apache.zookeeper.server.NettyServerCnxnFactory" >>> level="ERROR" /> >>> <logger name="org.apache.zookeeper.server.quorum" level="ERROR" /> >>> <logger name="org.apache.zookeeper.ZooKeeper" level="ERROR" /> >>> <logger name="org.apache.zookeeper.server.PrepRequestProcessor" >>> level="ERROR" /> >>> <logger name="org.apache.nifi.controller.reporting.LogComponentStatuses" >>> level="ERROR" /> >>> >>> <logger name="org.apache.calcite.runtime.CalciteException" level="OFF" /> >>> >>> <logger >>> name="org.apache.curator.framework.recipes.leader.LeaderSelector" >>> level="OFF" /> >>> <logger name="org.apache.curator.ConnectionState" level="OFF" /> >>> >>> <!-- Logger for managing logging statements for nifi clusters. --> >>> <logger name="org.apache.nifi.cluster" level="INFO"/> >>> >>> <!-- Logger for logging HTTP requests received by the web server. --> >>> <logger name="org.apache.nifi.server.JettyServer" level="INFO"/> >>> >>> <!-- Logger for managing logging statements for jetty --> >>> <logger name="org.eclipse.jetty" level="INFO"/> >>> >>> <!-- Suppress non-error messages due to excessive logging by class or >>> library --> >>> <logger name="org.springframework" level="ERROR"/> >>> >>> <!-- Suppress non-error messages due to known warning about redundant >>> path annotation (NIFI-574) --> >>> <logger name="org.glassfish.jersey.internal.Errors" level="ERROR"/> >>> >>> <!-- Suppress non-error messages due to Jetty AnnotationParser emitting >>> a large amount of WARNS. Issue described in NIFI-5479. --> >>> <logger name="org.eclipse.jetty.annotations.AnnotationParser" >>> level="ERROR"/> >>> >>> <!-- Suppress non-error messages from SSHJ which was emitting large >>> amounts of INFO logs by default --> >>> <logger name="net.schmizz.sshj" level="WARN" /> >>> <logger name="com.hierynomus.sshj" level="WARN" /> >>> >>> <!-- Suppress non-error messages from SMBJ which was emitting large >>> amounts of INFO logs by default --> >>> <logger name="com.hierynomus.smbj" level="WARN" /> >>> >>> <!-- Suppress non-error messages from AWS KCL which was emitting large >>> amounts of INFO logs by default --> >>> <logger name="com.amazonaws.services.kinesis" level="WARN" /> >>> >>> <!-- Suppress non-error messages from Apache Atlas which was emitting >>> large amounts of INFO logs by default --> >>> <logger name="org.apache.atlas" level="WARN" /> >>> >>> <logger name="org.apache.nifi.web.security.requests" level="INFO" /> >>> >>> <!-- >>> Logger for capturing user events. We do not want to propagate these >>> log events to the root logger. These messages are only sent to the >>> user-log appender. >>> --> >>> <logger name="org.apache.nifi.web.security" level="INFO" /> >>> <logger name="org.apache.nifi.web.api.config" level="INFO" /> >>> <logger name="org.apache.nifi.authorization" level="INFO" /> >>> <logger name="org.apache.nifi.cluster.authorization" level="INFO" /> >>> <logger name="org.apache.nifi.web.api.AccessResource" level="INFO" /> >>> <logger name="org.springframework.security.saml.log" level="WARN" /> >>> <logger name="org.opensaml" level="WARN" /> >>> <logger name="org.apache.nifi.web.server.RequestLog" level="INFO" /> >>> <logger name="org.apache.nifi.bootstrap" level="INFO" /> >>> <logger name="org.apache.nifi.bootstrap.Command" level="INFO" /> >>> >>> <logger name="org.apache.nifi.StdOut" level="INFO" /> >>> <logger name="org.apache.nifi.StdErr" level="INFO" /> >>> >>> <root level="INFO"> >>> <appender-ref ref="CONSOLE" /> >>> </root> >>> >>> </configuration> >>> >>> >>>
