Kevin, thank you for taking a look. I am using a custom Dockerfile (and
docker-compose.yaml) I wrote, pasted at the end of this email.
I also ran NiFi natively, without Docker, on my desktop computer and observed
identical results. I initially suspected Docker could somehow be affecting
stdout somehow, but in testing I've not found any evidence of that.
Dockerfile
FROM openjdk:11
# These are environment variables that are not meant to be
# configurable, they are essential to the operation of the container.
ENV NIFI_HOME /opt/nifi/
ENV NIFI_LOG_DIR /persistent-storage/logs
ENV NIFI_OVERRIDE_NIFIENV true
WORKDIR /opt
# Unpack NiFi and Toolkit
COPY nifi.zip /opt
COPY nifi-toolkit.zip /opt
RUN unzip nifi.zip
RUN unzip nifi-toolkit.zip
RUN mv nifi-1.17.0 nifi
RUN mv nifi-toolkit-1.17.0 nifi-toolkit
# Clean out unused files
RUN rm --force nifi.tar.gz nifi-toolkit.tar.gz
RUN rm --force /opt/nifi/bin/*.bat
RUN rm --force /opt/nifi/conf/*
RUN rm --force /opt/nifi-toolkit/*.bat
RUN mkdir /persistent-storage
WORKDIR /opt/nifi
# Set configuration
COPY bootstrap.conf bootstrap-notification-services.xml authorizers.xml
state-management.xml logback.xml /opt/nifi/conf/
COPY initialize.sh /opt/nifi/bin/
COPY postgres-jdbc-driver.jar /opt
COPY snowflake-jdbc-driver.jar /opt
COPY ZIP_codes_to_states.csv /opt
CMD ["/opt/nifi/bin/nifi.sh", "run"]
EXPOSE 8000/tcp
docker-compose.yaml
services:
nifi:
container_name: nifi
build: .
image: nifi
volumes: [/persistent-storage/:/persistent-storage/]
ports: [8000:8000]
environment: [INITIAL_ADMIN_IDENTITY:'*redacted*']
logging:
driver: "awslogs"
options:
awslogs-region: "*redacted*"
awslogs-group: "*redacted*"
awslogs-stream: "All logs"
________________________________
From: Kevin Doran <[email protected]>
Sent: Wednesday, September 28, 2022 9:48 AM
To: [email protected] <[email protected]>
Subject: Re: Trouble configuring logging
[EXTERNAL]
Dylan - I looked into this and am yet unable to offer an explaination. Perhaps
others that are familiar with how org.apache.nifi.StdOut can shed some light,
or else I will keep digging when I have a block of time. To help in my
understanding: Which Docker image are you using? Is it the apace/nifi image or
a custom one, and if custom, can you share the Dockerfile?
Thanks,
Kevin
On Sep 27, 2022 at 10:21:12, Dylan Klomparens
<[email protected]<mailto:[email protected]>> wrote:
I am attempting to configure logging for NiFi. I have NiFi running in a Docker
container, which sends all console logs to AWS CloudWatch. Therefore, I am
configuring NiFi to send all logs to the console.
The problem is, for some reason all log messages are coming from the
org.apache.nifi.StdOut logger. I cannot figure out why, since I would like
messages to be printed directly from the logger that is receiving them.
It seems like messages are "passing through" loggers, which are ultimately
printed out from the org.apache.nifi.StdOut logger. Here is an example of one
log message:
2022-09-27 10:08:01,849 INFO [NiFi logging handler] org.apache.nifi.StdOut
2022-09-27 10:08:01,848 INFO [pool-6-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
Repository
Why would every single log message come from the StdOut logger? And how can I
have logs delivered from the logger they're supposedly originally coming from?
My logback.xml configuration is below for reference.
<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="true">
<shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook" />
<contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
<resetJUL>true</resetJUL>
</contextListener>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder
class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
</encoder>
</appender>
<logger name="org.apache.nifi" level="INFO"/>
<logger name="org.apache.nifi.processors" level="WARN"/>
<logger name="org.apache.nifi.processors.standard.LogAttribute"
level="INFO"/>
<logger name="org.apache.nifi.processors.standard.LogMessage"
level="INFO"/>
<logger
name="org.apache.nifi.controller.repository.StandardProcessSession"
level="WARN" />
<logger name="org.apache.zookeeper.ClientCnxn" level="ERROR" />
<logger name="org.apache.zookeeper.server.NIOServerCnxn" level="ERROR" />
<logger name="org.apache.zookeeper.server.NIOServerCnxnFactory"
level="ERROR" />
<logger name="org.apache.zookeeper.server.NettyServerCnxnFactory"
level="ERROR" />
<logger name="org.apache.zookeeper.server.quorum" level="ERROR" />
<logger name="org.apache.zookeeper.ZooKeeper" level="ERROR" />
<logger name="org.apache.zookeeper.server.PrepRequestProcessor"
level="ERROR" />
<logger name="org.apache.nifi.controller.reporting.LogComponentStatuses"
level="ERROR" />
<logger name="org.apache.calcite.runtime.CalciteException" level="OFF" />
<logger name="org.apache.curator.framework.recipes.leader.LeaderSelector"
level="OFF" />
<logger name="org.apache.curator.ConnectionState" level="OFF" />
<!-- Logger for managing logging statements for nifi clusters. -->
<logger name="org.apache.nifi.cluster" level="INFO"/>
<!-- Logger for logging HTTP requests received by the web server. -->
<logger name="org.apache.nifi.server.JettyServer" level="INFO"/>
<!-- Logger for managing logging statements for jetty -->
<logger name="org.eclipse.jetty" level="INFO"/>
<!-- Suppress non-error messages due to excessive logging by class or
library -->
<logger name="org.springframework" level="ERROR"/>
<!-- Suppress non-error messages due to known warning about redundant
path annotation (NIFI-574) -->
<logger name="org.glassfish.jersey.internal.Errors" level="ERROR"/>
<!-- Suppress non-error messages due to Jetty AnnotationParser emitting a
large amount of WARNS. Issue described in NIFI-5479. -->
<logger name="org.eclipse.jetty.annotations.AnnotationParser"
level="ERROR"/>
<!-- Suppress non-error messages from SSHJ which was emitting large
amounts of INFO logs by default -->
<logger name="net.schmizz.sshj" level="WARN" />
<logger name="com.hierynomus.sshj" level="WARN" />
<!-- Suppress non-error messages from SMBJ which was emitting large
amounts of INFO logs by default -->
<logger name="com.hierynomus.smbj" level="WARN" />
<!-- Suppress non-error messages from AWS KCL which was emitting large
amounts of INFO logs by default -->
<logger name="com.amazonaws.services.kinesis" level="WARN" />
<!-- Suppress non-error messages from Apache Atlas which was emitting
large amounts of INFO logs by default -->
<logger name="org.apache.atlas" level="WARN" />
<logger name="org.apache.nifi.web.security.requests" level="INFO" />
<!--
Logger for capturing user events. We do not want to propagate these
log events to the root logger. These messages are only sent to the
user-log appender.
-->
<logger name="org.apache.nifi.web.security" level="INFO" />
<logger name="org.apache.nifi.web.api.config" level="INFO" />
<logger name="org.apache.nifi.authorization" level="INFO" />
<logger name="org.apache.nifi.cluster.authorization" level="INFO" />
<logger name="org.apache.nifi.web.api.AccessResource" level="INFO" />
<logger name="org.springframework.security.saml.log" level="WARN" />
<logger name="org.opensaml" level="WARN" />
<logger name="org.apache.nifi.web.server.RequestLog" level="INFO" />
<logger name="org.apache.nifi.bootstrap" level="INFO" />
<logger name="org.apache.nifi.bootstrap.Command" level="INFO" />
<logger name="org.apache.nifi.StdOut" level="INFO" />
<logger name="org.apache.nifi.StdErr" level="INFO" />
<root level="INFO">
<appender-ref ref="CONSOLE" />
</root>
</configuration>
