Yeah, if it helps at all, this is how we work around this in the Apache
Dockerfile (I knew realize why this method was used rather than a standard
console appender in logback):
# Continuously provide logs so that 'docker logs' can produce them
"${NIFI_HOME}/bin/nifi.sh" run &
nifi_pid="$!"
tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" &
https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129
Something to look into. As more NiFi deployments become containerized, this
may be an area for improvement over time.
Thanks,
Kevin
On Sep 28, 2022 at 19:40:35, Dylan Klomparens <[email protected]>
wrote:
> Mark and Kevin, thank you for your insightful comments. That information
> allowed me to piece together the puzzle. Indeed, anything that is sent to
> standard out subsequently *causes* the log message to come from the the
> org.apache.nifi.StdOut logger in the RunNiFi class. Your description
> allowed me to find the exact line of code that does this
> <https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485>
> .
>
> From my perspective this is an unfortunate design choice for NiFi because
> it does not allow standard out to be redirected easily. (For example,
> flowing onward to Docker, and forwarded to AWS CloudWatch using Docker's
> built-in log driver, in my case). I suppose I'll have to find an
> alternative logback appender class to output the logs to, then find a way
> to forward them on from there.
>
> Thanks again for the additional information that put the picture into view.
> ------------------------------
> *From:* Mark Payne <[email protected]>
> *Sent:* Wednesday, September 28, 2022 10:14 AM
> *To:* users <[email protected]>
> *Subject:* Re: Trouble configuring logging
>
>
> [EXTERNAL]
> This is because of how NiFi is run. When you startup nifi (bin/nifi.sh
> start) it doesn’t directly start the NiFi process.
> Instead, it starts a different processor, which is called RunNiFi. This
> RunNiFi process is responsible for doing a lot of things, including
> monitoring the nifi process and if it dies restarting it.
> Anything written to NiFi’s Standard Out goes to this processor, which then
> logs it.
> So you’d probably need to update the logging for the appender writing to
> the bootstrap file:
> <appender name="BOOTSTRAP_FILE"
> class="ch.qos.logback.core.rolling.RollingFileAppender”>
>
> And redirect that to standard out
>
> Thanks
> -Mark
>
>
> On Sep 28, 2022, at 9:48 AM, Kevin Doran <[email protected]> wrote:
>
> Dylan - I looked into this and am yet unable to offer an explaination.
> Perhaps others that are familiar with how org.apache.nifi.StdOut can shed
> some light, or else I will keep digging when I have a block of time. To
> help in my understanding: Which Docker image are you using? Is it the
> apace/nifi image or a custom one, and if custom, can you share the
> Dockerfile?
>
> Thanks,
> Kevin
>
> On Sep 27, 2022 at 10:21:12, Dylan Klomparens <[email protected]>
> wrote:
>
> I am attempting to configure logging for NiFi. I have NiFi running in a
> Docker container, which sends all console logs to AWS CloudWatch.
> Therefore, I am configuring NiFi to send all logs to the console.
>
> The problem is, for some reason *all log messages are coming from the
> org.apache.nifi.StdOut logger*. I cannot figure out why, since I would
> like messages to be printed directly from the logger that is receiving them.
>
> It seems like messages are "passing through" loggers, which are ultimately
> printed out from the org.apache.nifi.StdOut logger. Here is an example of
> *one* log message:
> *2022-09-27 10:08:01,849 INFO [NiFi logging handler]
> org.apache.nifi.StdOut* *2022-09-27 10:08:01,848 INFO [pool-6-thread-1]
> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
> Repository*
>
> *Why would every single log message come from the StdOut logger? And how
> can I have logs delivered from the logger they're supposedly originally
> coming from?*
>
> My logback.xml configuration is below for reference.
>
> <?xml version="1.0" encoding="UTF-8"?>
> <configuration debug="true">
> <shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook" />
>
> <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
> <resetJUL>true</resetJUL>
> </contextListener>
>
> <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
> <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
> </encoder>
> </appender>
>
> <logger name="org.apache.nifi" level="INFO"/>
> <logger name="org.apache.nifi.processors" level="WARN"/>
> <logger name="org.apache.nifi.processors.standard.LogAttribute"
> level="INFO"/>
> <logger name="org.apache.nifi.processors.standard.LogMessage"
> level="INFO"/>
> <logger
> name="org.apache.nifi.controller.repository.StandardProcessSession"
> level="WARN" />
>
> <logger name="org.apache.zookeeper.ClientCnxn" level="ERROR" />
> <logger name="org.apache.zookeeper.server.NIOServerCnxn" level="ERROR" />
> <logger name="org.apache.zookeeper.server.NIOServerCnxnFactory"
> level="ERROR" />
> <logger name="org.apache.zookeeper.server.NettyServerCnxnFactory"
> level="ERROR" />
> <logger name="org.apache.zookeeper.server.quorum" level="ERROR" />
> <logger name="org.apache.zookeeper.ZooKeeper" level="ERROR" />
> <logger name="org.apache.zookeeper.server.PrepRequestProcessor"
> level="ERROR" />
> <logger name="org.apache.nifi.controller.reporting.LogComponentStatuses"
> level="ERROR" />
>
> <logger name="org.apache.calcite.runtime.CalciteException" level="OFF" />
>
> <logger name="org.apache.curator.framework.recipes.leader.LeaderSelector"
> level="OFF" />
> <logger name="org.apache.curator.ConnectionState" level="OFF" />
>
> <!-- Logger for managing logging statements for nifi clusters. -->
> <logger name="org.apache.nifi.cluster" level="INFO"/>
>
> <!-- Logger for logging HTTP requests received by the web server. -->
> <logger name="org.apache.nifi.server.JettyServer" level="INFO"/>
>
> <!-- Logger for managing logging statements for jetty -->
> <logger name="org.eclipse.jetty" level="INFO"/>
>
> <!-- Suppress non-error messages due to excessive logging by class or
> library -->
> <logger name="org.springframework" level="ERROR"/>
>
> <!-- Suppress non-error messages due to known warning about redundant path
> annotation (NIFI-574) -->
> <logger name="org.glassfish.jersey.internal.Errors" level="ERROR"/>
>
> <!-- Suppress non-error messages due to Jetty AnnotationParser emitting a
> large amount of WARNS. Issue described in NIFI-5479. -->
> <logger name="org.eclipse.jetty.annotations.AnnotationParser"
> level="ERROR"/>
>
> <!-- Suppress non-error messages from SSHJ which was emitting large
> amounts of INFO logs by default -->
> <logger name="net.schmizz.sshj" level="WARN" />
> <logger name="com.hierynomus.sshj" level="WARN" />
>
> <!-- Suppress non-error messages from SMBJ which was emitting large
> amounts of INFO logs by default -->
> <logger name="com.hierynomus.smbj" level="WARN" />
>
> <!-- Suppress non-error messages from AWS KCL which was emitting large
> amounts of INFO logs by default -->
> <logger name="com.amazonaws.services.kinesis" level="WARN" />
>
> <!-- Suppress non-error messages from Apache Atlas which was emitting
> large amounts of INFO logs by default -->
> <logger name="org.apache.atlas" level="WARN" />
>
> <logger name="org.apache.nifi.web.security.requests" level="INFO" />
>
> <!--
> Logger for capturing user events. We do not want to propagate these
> log events to the root logger. These messages are only sent to the
> user-log appender.
> -->
> <logger name="org.apache.nifi.web.security" level="INFO" />
> <logger name="org.apache.nifi.web.api.config" level="INFO" />
> <logger name="org.apache.nifi.authorization" level="INFO" />
> <logger name="org.apache.nifi.cluster.authorization" level="INFO" />
> <logger name="org.apache.nifi.web.api.AccessResource" level="INFO" />
> <logger name="org.springframework.security.saml.log" level="WARN" />
> <logger name="org.opensaml" level="WARN" />
> <logger name="org.apache.nifi.web.server.RequestLog" level="INFO" />
> <logger name="org.apache.nifi.bootstrap" level="INFO" />
> <logger name="org.apache.nifi.bootstrap.Command" level="INFO" />
>
> <logger name="org.apache.nifi.StdOut" level="INFO" />
> <logger name="org.apache.nifi.StdErr" level="INFO" />
>
> <root level="INFO">
> <appender-ref ref="CONSOLE" />
> </root>
>
> </configuration>
>
>
>
